Generation of a continuous security monitoring evaluation regarding a system aspect of a system

ABSTRACT

A method includes determining, by an analysis system, a system aspect of a system for a continuous security monitoring evaluation. The method further includes determining, by the analysis system, at least one evaluation perspective and at least one evaluation viewpoint for use in performing the continuous security monitoring evaluation on the system aspect. The method further includes obtaining, by the analysis system, continuous security monitoring data regarding the system aspect in accordance with the at least one evaluation perspective and the at least one evaluation viewpoint. The method further includes calculating, by the analysis system, a continuous security monitoring rating as a measure of system continuous security monitoring maturity for the system aspect based on the continuous security monitoring data, the at least one evaluation perspective, the at least one evaluation viewpoint, and at least one evaluation rating metric.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility Patent application claims priority pursuant to35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No.17/247,701, entitled “GENERATION OF AN ISSUE DETECTION EVALUATIONREGARDING A SYSTEM ASPECT OF A SYSTEM,” filed Dec. 21, 2020, whichclaims priority pursuant to 35 U.S.C. § 119(e) to U.S. ProvisionalApplication No. 62/992,661, entitled “SYSTEM ANALYSIS SYSTEM,” filedMar. 20, 2020, both of which are hereby incorporated herein by referencein their entirety and made part of the present U.S. Utility PatentApplication for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable.

BACKGROUND OF THE INVENTION Technical Field of the Invention

This disclosure relates to computer systems and more particularly toevaluation of a computer system.

Description of Related Art

The structure and operation of the Internet and other publicly availablenetworks are well known and support computer systems (systems) ofmultitudes of companies, organizations, and individuals. A typicalsystem includes networking equipment, end point devices such as computerservers, user computers, storage devices, printing devices, securitydevices, and point of service devices, among other types of devices. Thenetworking equipment includes routers, switches, edge devices, wirelessaccess points, and other types of communication devices that intercouplein a wired or wireless fashion. The networking equipment facilitates thecreation of one or more networks that are tasked to service all or aportion of a company's communication needs, e.g., Wide Area Networks,Local Area Networks, Virtual Private Networks, etc.

Each device within a system includes hardware components and softwarecomponents. Hardware components degrade over time and eventually areincapable of performing their intended functions. Software componentsmust be updated regularly to ensure their proper functionality. Somesoftware components are simply replaced by newer and better softwareeven though they remain operational within a system.

Many companies and larger organizations have their own InformationTechnology (IT) departments. Others outsource their IT needs to thirdparty providers. The knowledge requirements for servicing a systemtypically outstrip the abilities of the IT department or third-partyprovider. Thus, hardware and software may not be functioning properlyand can adversely affect the overall system.

Cyber-attacks are initiated by individuals or entities with the badintent of stealing sensitive information such as login/passwordinformation, stealing proprietary information such as trade secrets orimportant new technology, interfering with the operation of a system,and/or holding the system hostage until a ransom is paid, among otherimproper purposes. A single cyber-attack can make a large systeminoperable and cost the system owner many millions of dollars to restoreand remedy.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee

FIG. 1 is a schematic block diagram of an embodiment of a networkedenvironment that includes systems coupled to an analysis system inaccordance with the present disclosure;

FIGS. 2A-2D are schematic block diagrams of embodiments of a computingdevice in accordance with the present disclosure;

FIGS. 3A-3E are schematic block diagrams of embodiments of a computingentity in accordance with the present disclosure;

FIG. 4 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system coupled to an analysis system inaccordance with the present disclosure;

FIG. 5 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system coupled to an analysis system inaccordance with the present disclosure;

FIG. 6 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system coupled to an analysis system inaccordance with the present disclosure;

FIG. 7 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system coupled to an analysis system inaccordance with the present disclosure;

FIG. 8 is a schematic block diagram of another embodiment of a networkedenvironment having a system that includes a plurality of system elementsin accordance with the present disclosure;

FIG. 9 is a schematic block diagram of an example of a system section ofa system selected for evaluation in accordance with the presentdisclosure;

FIG. 10 is a schematic block diagram of another example of a systemsection of a system selected for evaluation in accordance with thepresent disclosure;

FIG. 11 is a schematic block diagram of an embodiment of a networkedenvironment having a system that includes a plurality of system assetscoupled to an analysis system in accordance with the present disclosure;

FIG. 12 is a schematic block diagram of an embodiment of a system thatincludes a plurality of physical assets coupled to an analysis system inaccordance with the present disclosure;

FIG. 13 is a schematic block diagram of another embodiment of anetworked environment having a system that includes a plurality ofsystem assets coupled to an analysis system in accordance with thepresent disclosure;

FIG. 14 is a schematic block diagram of another embodiment of a systemthat includes a plurality of physical assets coupled to an analysissystem in accordance with the present disclosure;

FIG. 15 is a schematic block diagram of another embodiment of a systemthat includes a plurality of physical assets coupled to an analysissystem in accordance with the present disclosure;

FIG. 16 is a schematic block diagram of another embodiment of a systemthat includes a plurality of physical assets in accordance with thepresent disclosure;

FIG. 17 is a schematic block diagram of an embodiment of a usercomputing device in accordance with the present disclosure;

FIG. 18 is a schematic block diagram of an embodiment of a server inaccordance with the present disclosure;

FIG. 19 is a schematic block diagram of another embodiment of anetworked environment having a system that includes a plurality ofsystem functions coupled to an analysis system in accordance with thepresent disclosure;

FIG. 20 is a schematic block diagram of another embodiment of a systemthat includes divisions, departments, and groups in accordance with thepresent disclosure;

FIG. 21 is a schematic block diagram of another embodiment of a systemthat includes divisions and departments, which include system elementsin accordance with the present disclosure;

FIG. 22 is a schematic block diagram of another embodiment of a divisionof a system having departments, which include system elements inaccordance with the present disclosure;

FIG. 23 is a schematic block diagram of another embodiment of anetworked environment having a system that includes a plurality ofsecurity functions coupled to an analysis system in accordance with thepresent disclosure;

FIG. 24 is a schematic block diagram of an embodiment an engineeringdepartment of a division that reports to a corporate department of asystem in accordance with the present disclosure;

FIG. 25 is a schematic block diagram of an example of an analysis systemevaluating a system element under test of a system in accordance withthe present disclosure;

FIG. 26 is a schematic block diagram of another example of an analysissystem evaluating a system element under test of a system in accordancewith the present disclosure;

FIG. 27 is a schematic block diagram of another example of an analysissystem evaluating a system element under test of a system in accordancewith the present disclosure;

FIG. 28 is a schematic block diagram of another example of an analysissystem evaluating a system element under test of a system in accordancewith the present disclosure;

FIG. 29 is a schematic block diagram of an example of the functioning ofan analysis system evaluating a system element under test of a system inaccordance with the present disclosure;

FIG. 30 is a schematic block diagram of another example of thefunctioning of an analysis system evaluating a system element under testof a system in accordance with the present disclosure;

FIG. 31 is a diagram of an example of evaluation options of an analysissystem for evaluating a system element under test of a system inaccordance with the present disclosure;

FIG. 32 is a diagram of another example of evaluation options of ananalysis system for evaluating a system element under test of a systemin accordance with the present disclosure;

FIG. 33 is a diagram of another example of evaluation options of ananalysis system for evaluating a system element under test of a systemin accordance with the present disclosure;

FIG. 34 is a diagram of another example of evaluation options of ananalysis system for evaluating a system element under test of a systemin accordance with the present disclosure;

FIG. 35 is a schematic block diagram of an embodiment of an analysissystem coupled to a system in accordance with the present disclosure;

FIG. 36 is a schematic block diagram of an embodiment of a portion of ananalysis system coupled to a system in accordance with the presentdisclosure;

FIG. 37 is a schematic block diagram of another embodiment of a portionof an analysis system coupled to a system in accordance with the presentdisclosure;

FIG. 38 is a schematic block diagram of an embodiment of a dataextraction module of an analysis system coupled to a system inaccordance with the present disclosure;

FIG. 39 is a schematic block diagram of another embodiment of ananalysis system coupled to a system in accordance with the presentdisclosure;

FIG. 40 is a schematic block diagram of another embodiment of ananalysis system coupled to a system in accordance with the presentdisclosure;

FIG. 41 is a schematic block diagram of an embodiment of a data analysismodule of an analysis system in accordance with the present disclosure;

FIG. 42 is a schematic block diagram of an embodiment of an analyze andscore module of an analysis system in accordance with the presentdisclosure;

FIG. 43 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system for analyzing a section of a system in accordancewith the present disclosure;

FIG. 44 is a diagram of another example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system for analyzing a section of a system in accordancewith the present disclosure;

FIG. 45 is a diagram of an example of an identification evaluationcategory, sub-categories, and sub-sub-categories of the evaluationaspects and in accordance with the present disclosure;

FIG. 46 is a diagram of an example of a protect evaluation category,sub-categories, and sub-sub-categories of the evaluation aspects and inaccordance with the present disclosure;

FIG. 47 is a diagram of an example of a detect evaluation category,sub-categories, and sub-sub-categories of the evaluation aspects and inaccordance with the present disclosure;

FIG. 48 is a diagram of an example of a respond evaluation category,sub-categories, and sub-sub-categories of the evaluation aspects and inaccordance with the present disclosure;

FIG. 49 is a diagram of an example of a recover evaluation category,sub-categories, and sub-sub-categories of the evaluation aspects and inaccordance with the present disclosure;

FIG. 50 is a diagram of a specific example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system for analyzing a section of a system in accordancewith the present disclosure;

FIG. 51 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for analyzing a section of a systemin accordance with the present disclosure;

FIG. 52 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for analyzing a section of a systemin accordance with the present disclosure;

FIG. 53 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for analyzing a section of a systemin accordance with the present disclosure;

FIG. 54 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for analyzing a section of a systemin accordance with the present disclosure;

FIG. 55 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for analyzing a section of a systemin accordance with the present disclosure;

FIG. 56 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for analyzing a section of a systemin accordance with the present disclosure;

FIG. 57 is a diagram of an example of identifying deficiencies andauto-corrections by an analysis system analyzing a section of a systemin accordance with the present disclosure;

FIG. 58 is a schematic block diagram of an embodiment of an evaluationprocessing module of an analysis system in accordance with the presentdisclosure;

FIG. 59 is a state diagram of an example of an analysis system analyzinga section of a system in accordance with the present disclosure;

FIG. 60 is a logic diagram of an example of an analysis system analyzinga section of a system in accordance with the present disclosure;

FIG. 61 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 62 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 63 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 64 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 65 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 66 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 67 is a logic diagram of another example of an analysis systemanalyzing a section of a system in accordance with the presentdisclosure;

FIG. 68 is a logic diagram of an example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 69 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 70 is a schematic block diagram of an example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 71 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system for generating an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 72 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 73 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 74 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 75 is a diagram of an example of issue detection data for use by ananalysis system to generate an issue detection rating for a section of asystem in accordance with the present disclosure;

FIG. 76 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 77 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 78 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 79 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 80 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 81 is a schematic block diagram of an embodiment of a data analysismodule of an analysis system in accordance with the present disclosure;

FIG. 82 is a schematic block diagram of another embodiment of a dataanalysis module of an analysis system in accordance with the presentdisclosure;

FIG. 83 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 84 is a logic diagram of a further example of generating a processrating for understanding of system build for issue detection securityfunctions of an organization in accordance with the present disclosure;

FIG. 85 is a logic diagram of a further example of generating a processrating for understanding of verifying issue detection security functionsof an organization in accordance with the present disclosure;

FIG. 86 is a diagram of an example of issue detection data for use by ananalysis system to generate an issue detection rating for a section of asystem in accordance with the present disclosure;

FIG. 87 is a logic diagram of an example of generating a process ratingby the analysis system in accordance with the present disclosure;

FIG. 88 is a logic diagram of an example of generating a process ratingby the analysis system in accordance with the present disclosure;

FIG. 89 is a logic diagram of an example of generating a process ratingby the analysis system in accordance with the present disclosure;

FIG. 90 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 91 is a logic diagram of a further example of generating a policyrating for understanding of system build for issue detection securityfunctions of an organization in accordance with the present disclosure;

FIG. 92 is a logic diagram of a further example of generating a policyrating for understanding of verifying issue detection security functionsof an organization in accordance with the present disclosure;

FIG. 93 is a logic diagram of an example of generating a policy ratingby the analysis system in accordance with the present disclosure;

FIG. 94 is a logic diagram of an example of generating a policy ratingby the analysis system in accordance with the present disclosure;

FIG. 95 is a logic diagram of an example of generating a policy ratingby the analysis system in accordance with the present disclosure;

FIG. 96 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 97 is a logic diagram of a further example of generating adocumentation rating for understanding of system build of issuedetection security functions of an organization in accordance with thepresent disclosure;

FIG. 98 is a logic diagram of a further example of generating adocumentation rating for understanding of system build of issuedetection security functions of an organization in accordance with thepresent disclosure;

FIG. 99 is a logic diagram of an example of generating a documentationrating by the analysis system in accordance with the present disclosure;

FIG. 100 is a logic diagram of an example of generating a documentationrating by the analysis system in accordance with the present disclosure;

FIG. 101 is a logic diagram of an example of generating a documentationrating by the analysis system in accordance with the present disclosure;

FIG. 102 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 103 is a logic diagram of a further example of generating anautomation rating for understanding of system build of issue detectionsecurity functions of an organization in accordance with the presentdisclosure;

FIG. 104 is a logic diagram of a further example of generating anautomation rating for understanding of system build of issue detectionsecurity functions of an organization in accordance with the presentdisclosure;

FIG. 105 is a logic diagram of an example of generating an automationrating by the analysis system in accordance with the present disclosure;

FIG. 106 is a logic diagram of an example of generating an automationrating by the analysis system in accordance with the present disclosure;

FIG. 107 is a logic diagram of an example of generating an automationrating by the analysis system in accordance with the present disclosure;

FIG. 108 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 109 is a diagram of another specific example of system aspects,evaluation aspects, evaluation rating metrics, and analysis systemoutput options of an analysis system for generating an issue detectionrating for a section of a system in accordance with the presentdisclosure;

FIG. 110 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 111 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 112 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asection of a system in accordance with the present disclosure;

FIG. 113 is a diagram of an example of combining one or more individualissue detection ratings into an issue detection rating in accordancewith the present disclosure;

FIG. 114 is a diagram of another example of combining one or moreindividual issue detection ratings into an issue detection rating inaccordance with the present disclosure;

FIG. 115 is a diagram of another example of combining one or moreindividual issue detection ratings into an issue detection rating inaccordance with the present disclosure;

FIG. 116 is a diagram of another example of combining one or moreindividual issue detection ratings into an issue detection rating inaccordance with the present disclosure;

FIG. 117 is a diagram of another example of combining one or moreindividual issue detection ratings into an issue detection rating inaccordance with the present disclosure;

FIG. 118 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 119 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 120 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 121 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 122 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 123 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 124 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 125 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 126 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 127 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 128 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 129 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 130 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 131 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 132 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 133 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure;

FIG. 134 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure; and

FIG. 135 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a section of a system inaccordance with the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic block diagram of an embodiment of a networkedenvironment that includes one or more networks 14, external data feedssources 15, a plurality of systems 11-13, and an analysis system 10. Theexternal data feed sources 15 includes one or more system proficiencyresources 22, one or more business associated computing devices 23, oneor more non-business associated computing devices 24 (e.g., publiclyavailable servers 27 and subscription based servers 28), one or more BOT(i.e., internet robot) computing devices 25, and one or more bad actorcomputing devices 26. The analysis system 10 includes one or moreanalysis computing entities 16, a plurality of analysis system modules17 (one or more in each of the systems 11-13), and a plurality ofstorage systems 19-21 (e.g., system A private storage 19, system Bprivate storage 20, through system x private storage 21, and otherstorage). Each of the systems 11-13 includes one or more networkinterfaces 18 and many more elements not shown in FIG. 1.

A computing device may be implemented in a variety of ways. A fewexamples are shown in FIGS. 2A-2D. A computing entity may be implementedin a variety of ways. A few examples are shown in FIGS. 3A-3E.

A storage system 19-21 may be implemented in a variety of ways. Forexample, each storage system is a standalone database. As anotherexample, the storage systems are implemented in a common database. Adatabase is a centralized database, a distributed database, anoperational database, a cloud database, an object-oriented database,and/or a relational database. A storage system 19-21 is coupled to theanalysis system 10 using a secure data pipeline to limit and controlaccess to the storage systems. The secure data pipeline may beimplemented in a variety of ways. For example, the secure data pipelineis implemented on a provide network of the analysis system and/or of asystem under test. As another example, the secure data pipeline isimplemented via the network 14 using access control, using networkcontrols, implementing access and control policies, using encryption,using data loss prevention tools, and/or using auditing tools.

The one or more networks 14 includes one or more wide area networks(WAN), one or more local area networks (LAN), one or more wireless LANs(WLAN), one or more cellular networks, one or more satellite networks,one or more virtual private networks (VPN), one or more campus areanetworks (CAN), one or more metropolitan area networks (MAN), one ormore storage area networks (SAN), one or more enterprise privatenetworks (EPN), and/or one or more other type of networks.

In general, a system proficiency resource 22 is a source for dataregarding best-in-class practices (for system requirements, for systemdesign, for system implementation, and/or for system operation),governmental and/or regulatory requirements, security risk awarenessand/or risk remediation information, security risk avoidance,performance optimization information, system development guidelines,software development guideline, hardware requirements, networkingrequirements, networking guidelines, and/or other system proficiencyguidance. “Framework for Improving Critical Instructure Cybersecurity”,Version 1.1, Apr. 16, 2018 by the National Institute of Standards andTechnology (NIST) is an example of a system proficiency in the form of aguideline for cybersecurity.

A business associated computing device 23 is one that is operated by abusiness associate of the system owner. Typically, the businessassociated computing device 23 has access to at least a limited portionof the system to which the general public does not have access. Forexample, the business associated computing device 23 is operated by avendor of the organization operating the system and is granted limitedaccess for order placement and/or fulfillment. As another example, thebusiness associated computing device 23 is operated by a customer of theorganization operating the system and is granted limited access forplacing orders.

A non-business associated computing device 24 is a computing deviceoperated by a person or entity that does not have a businessrelationship with the organization operating the system. Suchnon-business associated computing device 24 are not granted specialaccess to the system. For example, a non-business associated computingdevice 24 is a publicly available server 27 to which a user computingdevice of the system may access. As another example, a non-businessassociated computing device 24 is a subscription based servers 28 towhich a user computing device of the system may access if it isauthorized by a system administrator of the system to have asubscription and has a valid subscription. As yet another example, thenon-business associated computing device 24 is a computing deviceoperated by a person or business that does not have an affiliation withthe organization operating the system.

A bot (i.e., internet robot) computing device 25 is a computing devicethat runs, with little to no human interaction, to interact with asystem and/or a computing device of a user via the internet or anetwork. There are a variety of types of bots. For example, there aresocial media bots, chatbots, bot crawlers, transaction bots, informationbots, and entertainment bots (e.g., games, art, books, etc.).

A bad actor computing device 26 is a computing device operated by aperson whose use of the computing device is for illegal and/or immoralpurposes. The bad actor computing device 26 may employ a bot to executean illegal and/or immoral purpose. In addition or in the alternative,the person may instruct the bad actor computing device to perform theillegal and/or immoral purpose, such as hacking, planting a worm,planting a virus, stealing data, uploading false data, and so on.

The analysis system 10 is operable to evaluate a system 11-13, orportion thereof, in a variety of ways. For example, the analysis system10 evaluates system A 11, or a portion thereof, by testing theorganization's understanding of its system, or portion thereof bytesting the organization's implementation of its system, or portionthereof and/or by testing the system's, or portion thereof; operation.As a specific example, the analysis system 10 tests the organization'sunderstanding of its system requirements for the implementation and/oroperation of its system, or portion thereof. As another specificexample, the analysis system 10 tests the organization's understandingof its software maintenance policies and/or procedures. As anotherspecific example, the analysis system 10 tests the organization'sunderstanding of its cybersecurity policies and/or procedures.

There is an almost endless combination of ways in which the analysissystem 10 can evaluate a system 11-13, which may be a computer system, acomputer network, an enterprise system, and/or other type of system thatincludes computing devices operating software. For example, the analysissystem 10 evaluates a system aspect (e.g., the system or a portion ofit) based on an evaluation aspect (e.g., options for how the system, orportion thereof, can be evaluated) in view of evaluation rating metrics(e.g., how the system, or portion thereof, is evaluated) to produce ananalysis system output (e.g., an evaluation rating, deficiencyidentification, and/or deficiency auto-correction).

The system aspect (e.g., the system or a portion thereof) includes aselection of one or more system elements of the system, a selection ofone or more system criteria, and/or a selection of one or more systemmodes. A system element of the system includes one or more system assetswhich is one or more physical assets of the system and/or conceptualassets of the system. For example, a physical asset is a computingentity, a computing device, a user software application, a systemsoftware application (e.g., operating system, etc.), a software tool, anetwork software application, a security software application, a systemmonitoring software application, and the like. As another example, aconceptual asset is a hardware architectural layout, or portion thereof,and/or a software architectural layout, or portion thereof.

A system element and/or system asset may be identified in a variety ofways. For example, it is identifiable by its use and/or location withinthe organization. As a specific example, a system element and/or systemasset is identified by an organizational identifier, a division of theorganization identifier, a department of a division identifier, a groupof a department identifier, and/or a sub-group of a group identifier. Inthis manner, if the entire system is to be evaluated, the organizationidentifier is used to select all of the system elements in the system.If a portion of the system is to be test based on business function,then a division, department, group, and/or sub-group identifier is usedto select the desired portion of the system.

In addition or in the alternative, a system element and/or system assetis identifiable based on a serial number, an IP (internet protocol)address, a vendor name, a type of system element and/or system asset(e.g., computing entity, a particular user software application, etc.),registered user of the system element and/or system asset, and/or otheridentifying metric. In this manner, an individual system element and/orsystem asset can be evaluated and/or a type of system element and/orsystem asset can be evaluated (e.g., a particular user softwareapplication).

A system criteria is regarding a level of the system, or portionthereof, being evaluated. For example, the system criteria includesguidelines, system requirements, system design, system build, andresulting system. As a further example, the guidelines (e.g., businessobjectives, security objectives, NIST cybersecurity guidelines, systemobjectives, governmental and/or regulatory requirements, third partyrequirements, etc.) are used to develop the system requirements, whichare used to design the system, which is used to the build the resultingsystem. As such, the system, or portion thereof, can be evaluated from aguideline level, a system requirements level, a design level, a buildlevel, and/or a resulting system level.

A system mode is regarding a different level of the system, or portionthereof, being evaluated. For example, the system mode includes assets,system functions, and security functions. As such, the system can beevaluated from an assets level, a system function level, and/or asecurity function level.

The evaluation aspect (e.g., options for how the system, or portionthereof, can be evaluated) includes a selection of one or moreevaluation perspectives, a selection of one or more evaluationviewpoints, and/or a selection of one or more evaluation categories,which may further includes sub-categories, and sub-categories of thesub-categories). An evaluation perspective is understanding of thesystem, or portion thereof; implementation (e.g., design and build) ofthe system, or portion thereof operational performance of the system, orportion thereof; or self-analysis of the system, or portion thereof.

An evaluation viewpoint is disclosed information from the system,discovered information about the system by the analysis system, ordesired information about the system obtained by the analysis systemfrom system proficiency resources. The evaluation viewpoint complementsthe evaluation perspective to allow for more in-depth and/or detailedevaluations. For example, the analysis system 10 can evaluate how wellthe system is understood by comparing disclosed data with discovereddata. As another example, the analysis system 10 can evaluate how wellthe system is actually implemented in comparison to a desired level ofimplementation.

The evaluation category includes an identify category, a protectcategory, a detect category, a respond category, and a recover category.Each evaluation category includes a plurality of sub-categories and, atleast some of the sub-categories include their own sub-categories (e.g.,a sub-sub category). For example, the identify category includes thesub-categories of asset management, business environment, governance,risk assessment, risk management, access control, awareness & training,and data security. As a further example, asset management includes thesub-categories of hardware inventory, software inventory, data flowmaps, external system cataloged, resource prioritization, and securityroles. The analysis system 10 can evaluate the system, or portionthereof, in light of one more evaluation categories, in light of anevaluation category and one or more sub-categories, or in light of anevaluation category, a sub-category, and one or more sub-sub-categories.

The evaluation rating metrics (e.g., how the system, or portion thereof,is evaluated) includes a selection of process, policy, procedure,certification, documentation, and/or automation. This allows theanalysis system to quantify its evaluation. For example, the analysissystem 10 can evaluate the processes a system, or portion thereof, hasto generate an evaluation rating, to identify deficiencies, and/or toauto-correct deficiencies. As another example, the analysis system 10can evaluate how well the system, or portion thereof, uses the processit has to generate an evaluation rating, to identify deficiencies,and/or to auto-correct deficiencies.

In an example, the analysis computing entity 16 (which includes one ormore computing entities) sends a data gathering request to the analysissystem module 17. The data gathering request is specific to theevaluation to be performed by the analysis system 10. For example, ifthe analysis system 10 is evaluating the understanding of the policies,processes, documentation, and automation regarding the assets built forthe engineering department, then the data gathering request would bespecific to policies, processes, documentation, and automation regardingthe assets built for the engineering department.

The analysis system module 17 is loaded on the system 11-13 and obtainedthe requested data from the system. The obtaining of the data can bedone in a variety of ways. For example, the data is disclosed by one ormore system administrators. The disclosed data corresponds to theinformation the system administrator(s) has regarding the system. Inessence, the disclosed data is a reflection of the knowledge the systemadministrator(s) has regarding the system.

As another example, the analysis system module 17 communicates withphysical assets of the system to discover the data. The communicationmay be direct with an asset. For example, the analysis system module 17sends a request to a particular computing device. Alternatively or inaddition, the communication may be through one or more discovery toolsof the system. For example, the analysis system module 17 communicateswith one or more tools of the system to obtain data regarding datasegregation & boundary, infrastructure management, exploit & malwareprotection, encryption, identity & access management, system monitoring,vulnerability management, and/or data protection.

A tool is a network monitoring tool, a network strategy and planningtool, a network managing tool, a Simple Network Management Protocol(SNMP) tool, a telephony monitoring tool, a firewall monitoring tool, abandwidth monitoring tool, an IT asset inventory management tool, anetwork discovery tool, a network asset discovery tool, a softwarediscovery tool, a security discovery tool, an infrastructure discoverytool, Security Information & Event Management (SIEM) tool, a datacrawler tool, and/or other type of tool to assist in discovery ofassets, functions, security issues, implementation of the system, and/oroperation of the system. If the system does not have a particular tool,the analysis system module 17 engages one to discover a particular pieceof data.

The analysis system module 17 provides the gathered data to the analysiscomputing entity 16, which stores the gathered data in a private storage19-21 and processes it. The gathered data is processed alone, incombination with stored data (of the system being evaluated and/oranother system's data), in combination with desired data (e.g., systemproficiencies), in combination with analysis modeling (e.g., riskmodeling, data flow modeling, security modeling, etc.), and/or incombination with stored analytic data (e.g., results of otherevaluations). As a result of the processing, the analysis computingentity 16 produces an evaluation rating, to identify deficiencies,and/or to auto-correct deficiencies. The evaluation results are storedin a private storage and/or in another database.

The analysis system 10 is operable to evaluate a system and/or itseco-system at any level of granularity from the entire system to anindividual asset over a wide spectrum of evaluation options. As anexample, the evaluation is to test understanding of the system, to testthe implementation of the system, and/or to test the operation of thesystem. As another example, the evaluation is to test the system'sself-evaluation capabilities with respect to understanding,implementation, and/or operation. As yet another example, the evaluationis to test policies regarding software tools; to test which softwaretools are prescribed by policy; to test which software tools areprohibited by policy; to test the use of the software tools inaccordance with policy, to test maintenance of software tools inaccordance with policy; to test the sufficiency of the policies, to testthe effectiveness of the policies; and/or to test compliancy with thepolicies.

The analysis system 10 takes an outside perspective to analyze thesystem. From within the system, it is often difficult to test the entiresystem, to test different combinations of system elements, to identifyareas of vulnerabilities (assets and human operators), to identify areasof strength (assets and human operators), and to be proactive. Further,such evaluations are additional tasks the system has to perform, whichmeans it consumes resources (human, physicals assets, and financial).Further, since system analysis is not the primary function of a system(supporting the organization is the system's primary purpose), thesystem analysis is not as thoroughly developed, implemented, and/orexecuted as is possible when its implemented in a stand-alone analysissystem, like system 10.

The primary purpose of the analysis system is to analyze other systemsto determine an evaluation rating, to identify deficiencies in thesystem, and, where it can, auto-correct the deficiencies. The evaluationrating can be regarding how well the system, or portion thereof, isunderstood, how well it is implemented, and/or how well it operates. Theevaluation rating can be regarding how effective the system, or portionthereof, is believed (disclosed data) to support a business function;actually (discovered data) supports a business function; and/or should(desired data) support the business function.

The evaluation rating can be regarding how effective the system, orportion thereof, is believed (disclosed data) to mitigate securityrisks; actually (discovered data) supports mitigating security risks;and/or should (desired data) support mitigating security risks. Theevaluation rating can be regarding how effective the system, or portionthereof, is believed (disclosed data) to respond to security risks;actually (discovered data) supports responding to security risks; and/orshould (desired data) support responding security risks.

The evaluation rating can be regarding how effective the system, orportion thereof, is believed (disclosed data) to be used by people; isactually (discovered data) used by people; and/or should (desired data)be used by people. The evaluation rating can be regarding how effectivethe system, or portion thereof, is believed (disclosed data) to identifyassets of the system; actually (discovered data) identifies assets ofthe system; and/or should (desired data) identify assets of the system.

There are a significant number of combinations in which the analysissystem 10 can evaluate a system 11-13. A primary purpose the analysissystem 10 is help the system 11-13 become more self-healing, moreself-updating, more self-protecting, more self-recovering, moreself-evaluating, more self-aware, more secure, more efficient, moreadaptive, and/or more self-responding. By discovering the strengths,weaknesses, vulnerabilities, and other system limitations in a way thatthe system itself cannot do effectively, the analysis system 10significantly improves the usefulness, security, and efficiency ofsystems 11-13.

FIG. 2A is a schematic block diagram of an embodiment of a computingdevice 40 that includes a plurality of computing resources. Thecomputing resource include a core control module 41, one or moreprocessing modules 43, one or more main memories 45, a read only memory(ROM) 44 for a boot up sequence, cache memory 47, a video graphicsprocessing module 42, a display 48 (optional), an Input-Output (I/O)peripheral control module 46, an I/O interface module 49 (which could beomitted), one or more input interface modules 50, one or more outputinterface modules 51, one or more network interface modules 55, and oneor more memory interface modules 54. A processing module 43 is describedin greater detail at the end of the detailed description of theinvention section and, in an alternative embodiment, has a directionconnection to the main memory 45. In an alternate embodiment, the corecontrol module 41 and the I/O and/or peripheral control module 46 areone module, such as a chipset, a quick path interconnect (QPI), and/oran ultra-path interconnect (UPI).

Each of the main memories 45 includes one or more Random Access Memory(RAM) integrated circuits, or chips. For example, a main memory 45includes four DDR4 (4^(th) generation of double data rate) RAM chips,each running at a rate of 2,400 MHz. In general, the main memory 45stores data and operational instructions most relevant for theprocessing module 43. For example, the core control module 41coordinates the transfer of data and/or operational instructions betweenthe main memory 45 and the memory 56-57. The data and/or operationalinstructions retrieve from memory 56-57 are the data and/or operationalinstructions requested by the processing module or will most likely beneeded by the processing module. When the processing module is done withthe data and/or operational instructions in main memory, the corecontrol module 41 coordinates sending updated data to the memory 56-57for storage.

The memory 56-57 includes one or more hard drives, one or more solidstate memory chips, and/or one or more other large capacity storagedevices that, in comparison to cache memory and main memory devices,is/are relatively inexpensive with respect to cost per amount of datastored. The memory 56-57 is coupled to the core control module 41 viathe I/O and/or peripheral control module 46 and via one or more memoryinterface modules 54. In an embodiment, the I/O and/or peripheralcontrol module 46 includes one or more Peripheral Component Interface(PCI) buses to which peripheral components connect to the core controlmodule 41. A memory interface module 54 includes a software driver and ahardware connector for coupling a memory device to the I/O and/orperipheral control module 46. For example, a memory interface 54 is inaccordance with a Serial Advanced Technology Attachment (SATA) port.

The core control module 41 coordinates data communications between theprocessing module(s) 43 and the network(s) 14 via the I/O and/orperipheral control module 46, the network interface module(s) 55, and anetwork card 58 or 59. A network card 58 or 59 includes a wirelesscommunication unit or a wired communication unit. A wirelesscommunication unit includes a wireless local area network (WLAN)communication device, a cellular communication device, a Bluetoothdevice, and/or a ZigBee communication device. A wired communication unitincludes a Gigabit LAN connection, a Firewire connection, and/or aproprietary computer wired connection. A network interface module 55includes a software driver and a hardware connector for coupling thenetwork card to the I/O and/or peripheral control module 46. Forexample, the network interface module 55 is in accordance with one ormore versions of IEEE 802.11, cellular telephone protocols, 10/100/1000Gigabit LAN protocols, etc.

The core control module 41 coordinates data communications between theprocessing module(s) 43 and input device(s) 52 via the input interfacemodule(s) 50, the I/O interface 49, and the I/O and/or peripheralcontrol module 46. An input device 52 includes a keypad, a keyboard,control switches, a touchpad, a microphone, a camera, etc. An inputinterface module 50 includes a software driver and a hardware connectorfor coupling an input device to the I/O and/or peripheral control module46. In an embodiment, an input interface module 50 is in accordance withone or more Universal Serial Bus (USB) protocols.

The core control module 41 coordinates data communications between theprocessing module(s) 43 and output device(s) 53 via the output interfacemodule(s) 51 and the I/O and/or peripheral control module 46. An outputdevice 53 includes a speaker, auxiliary memory, headphones, etc. Anoutput interface module 51 includes a software driver and a hardwareconnector for coupling an output device to the I/O and/or peripheralcontrol module 46. In an embodiment, an output interface module 46 is inaccordance with one or more audio codec protocols.

The processing module 43 communicates directly with a video graphicsprocessing module 42 to display data on the display 48. The display 48includes an LED (light emitting diode) display, an LCD (liquid crystaldisplay), and/or other type of display technology. The display has aresolution, an aspect ratio, and other features that affect the qualityof the display. The video graphics processing module 42 receives datafrom the processing module 43, processes the data to produce rendereddata in accordance with the characteristics of the display, and providesthe rendered data to the display 48.

FIG. 2B is a schematic block diagram of an embodiment of a computingdevice 40 that includes a plurality of computing resources similar tothe computing resources of FIG. 2A with the addition of one or morecloud memory interface modules 60, one or more cloud processinginterface modules 61, cloud memory 62, and one or more cloud processingmodules 63. The cloud memory 62 includes one or more tiers of memory(e.g., ROM, volatile (RAM, main, etc.), non-volatile (hard drive,solid-state, etc.) and/or backup (hard drive, tape, etc.)) that isremoted from the core control module and is accessed via a network (WANand/or LAN). The cloud processing module 63 is similar to processingmodule 43 but is remoted from the core control module and is accessedvia a network.

FIG. 2C is a schematic block diagram of an embodiment of a computingdevice 40 that includes a plurality of computing resources similar tothe computing resources of FIG. 2B with a change in how the cloud memoryinterface module(s) 60 and the cloud processing interface module(s) 61are coupled to the core control module 41. In this embodiment, theinterface modules 60 and 61 are coupled to a cloud peripheral controlmodule 63 that directly couples to the core control module 41.

FIG. 2D is a schematic block diagram of an embodiment of a computingdevice 40 that includes a plurality of computing resources, whichincludes include a core control module 41, a boot up processing module66, boot up RAM 67, a read only memory (ROM) 45, a video graphicsprocessing module 42, a display 48 (optional), an Input-Output (I/O)peripheral control module 46, one or more input interface modules 50,one or more output interface modules 51, one or more cloud memoryinterface modules 60, one or more cloud processing interface modules 61,cloud memory 62, and cloud processing module(s) 63.

In this embodiment, the computing device 40 includes enough processingresources (e.g., module 66, ROM 44, and RAM 67) to boot up. Once bootedup, the cloud memory 62 and the cloud processing module(s) 63 functionas the computing device's memory (e.g., main and hard drive) andprocessing module.

FIG. 3A is schematic block diagram of an embodiment of a computingentity 16 that includes a computing device 40 (e.g., one of theembodiments of FIGS. 2A-2D). A computing device may function as a usercomputing device, a server, a system computing device, a data storagedevice, a data security device, a networking device, a user accessdevice, a cell phone, a tablet, a laptop, a printer, a game console, asatellite control box, a cable box, etc.

FIG. 3B is schematic block diagram of an embodiment of a computingentity 16 that includes two or more computing devices 40 (e.g., two ormore from any combination of the embodiments of FIGS. 2A-2D). Thecomputing devices 40 perform the functions of a computing entity in apeer processing manner (e.g., coordinate together to perform thefunctions), in a master-slave manner (e.g., one computing devicecoordinates and the other support it), and/or in another manner.

FIG. 3C is schematic block diagram of an embodiment of a computingentity 16 that includes a network of computing devices 40 (e.g., two ormore from any combination of the embodiments of FIGS. 2A-2D). Thecomputing devices are coupled together via one or more networkconnections (e.g., WAN, LAN, cellular data, WLAN, etc.) and preform thefunctions of the computing entity.

FIG. 3D is schematic block diagram of an embodiment of a computingentity 16 that includes a primary computing device (e.g., any one of thecomputing devices of FIGS. 2A-2D), an interface device (e.g., a networkconnection), and a network of computing devices 40 (e.g., one or morefrom any combination of the embodiments of FIGS. 2A-2D). The primarycomputing device utilizes the other computing devices as co-processorsto execute one or more the functions of the computing entity, as storagefor data, for other data processing functions, and/or storage purposes.

FIG. 3E is schematic block diagram of an embodiment of a computingentity 16 that includes a primary computing device (e.g., any one of thecomputing devices of FIGS. 2A-2D), an interface device (e.g., a networkconnection) 70, and a network of computing resources 71 (e.g., two ormore resources from any combination of the embodiments of FIGS. 2A-2D).The primary computing device utilizes the computing resources asco-processors to execute one or more the functions of the computingentity, as storage for data, for other data processing functions, and/orstorage purposes.

FIG. 4 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system 11 (or system 12 or system 13), theanalysis system 10, one or more networks, one or more system proficiencyresources 22, one or more business associated computing devices 23, oneor more non-business associated computing devices 24 (e.g., publiclyavailable servers 27 and subscription based servers 28), one or more BOTcomputing devices 25, and one or more bad actor computing devices 26.This diagram is similar to FIG. 1 with the inclusion of detail withinthe system proficiency resource(s) 22, with inclusion of detail withinthe system 11, and with the inclusion of detail within the analysissystem module 17.

In addition to the discussion with respect FIG. 1, a system proficiencyresource 22 is a computing device that provides information regardingbest-in-class assets, best-in-class practices, known protocols, leadingedge information, and/or established guidelines regarding riskassessment, devices, software, networking, data security, cybersecurity,and/or data communication. A system proficiency resource 22 is acomputing device that may also provide information regarding standards,information regarding compliance requirements, information regardinglegal requirements, and/or information regarding regulatoryrequirements.

The system 11 is shown to include three inter-dependent modes: systemfunctions 82, security functions 83, and system assets 84. Systemfunctions 82 correspond to the functions the system executes to supportthe organization's business requirements. Security functions 83correspond to the functions the system executes to support theorganization's security requirements. The system assets 84 are thehardware and/or software platforms that support system functions 82and/or the security functions 83.

The analysis system module 17 includes one or more data extractionmodules 80 and one or more system user interface modules 81. A dataextraction module 80, which will be described in greater detail withreference to one or more subsequent figures, gathers data from thesystem for analysis by the analysis system 10. A system user interfacemodule 81 provides a user interface between the system 11 and theanalysis system 10 and functions to provide user information to theanalysis system 10 and to receive output data from the analysis system.The system user interface module 81 will be described in greater detailwith reference to one or more subsequent figures.

FIG. 5 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system 11 (or system 12 or system 13), theanalysis system 10, one or more networks, one or more system proficiencyresources 22, one or more business associated computing devices 23, oneor more non-business associated computing devices 24 (e.g., publiclyavailable servers 27 and subscription based servers 28), one or more BOTcomputing devices 25, and one or more bad actor computing devices 26.This diagram is similar to FIG. 4 with the inclusion of additionaldetail within the system 11.

In this embodiment, the system 11 includes a plurality of sets of systemassets to support the system functions 82 and/or the security functions83. For example, a set of system assets supports the system functions 82and/or security functions 83 for a particular business segment (e.g., adepartment within the organization). As another example, a second set ofsystem assets supports the security functions 83 for a differentbusiness segment and a third set of system assets supports the systemfunctions 82 for the different business segment.

FIG. 6 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system 11 (or system 12 or system 13), theanalysis system 10, one or more networks, one or more system proficiencyresources 22, one or more business associated computing devices 23, oneor more non-business associated computing devices 24 (e.g., publiclyavailable servers 27 and subscription based servers 28), one or more BOTcomputing devices 25, and one or more bad actor computing devices 26.This diagram is similar to FIG. 5 with the inclusion of additionaldetail within the system 11.

In this embodiment, the system 11 includes a plurality of sets of systemassets 84, system functions 82, and security functions 83. For example,a set of system assets 84, system functions 82, and security functions83 supports one department in an organization and a second set of systemassets 84, system functions 82, and security functions 83 supportsanother department in the organization.

FIG. 7 is a schematic block diagram of another embodiment of a networkedenvironment that includes a system 11 (or system 12 or system 13), theanalysis system 10, one or more networks, one or more system proficiencyresources 22, one or more business associated computing devices 23, oneor more non-business associated computing devices 24 (e.g., publiclyavailable servers 27 and subscription based servers 28), one or more BOTcomputing devices 25, and one or more bad actor computing devices 26.This diagram is similar to FIG. 4 with the inclusion of additionaldetail within the system 11.

In this embodiment, the system 11 includes system assets 84, systemfunctions 82, security functions 83, and self-evaluation functions 85.The self-evaluation functions 85 are supported by the system assets 84and are used by the system to evaluate its assets, is system functions,and its security functions. In general, self-evaluates looks at system'sability to analyze itself for self-determining it's understanding(self-aware) of the system; self-determining the implementation of thesystem, and/or self-determining operation of the system. In addition,the self-evaluation may further consider the system's ability toself-heal, self-update, self-protect, self-recover, self-evaluate,and/or self-respond. The analysis system 10 can evaluate theunderstanding, implementation, and/or operation of the self-evaluationfunctions.

FIG. 8 is a schematic block diagram of another embodiment of a networkedenvironment having a system 11 (or system 12 or system 13), the analysissystem 10, one or more networks represented by networkinginfrastructure, one or more system proficiency resources 22, one or morebusiness associated computing devices 23, one or more publicly availableservers 27, one or more subscription based servers 28, one or more BOTcomputing devices 25, and one or more bad actor computing devices 26.

In this embodiment, the system 11 is shown to include a plurality ofphysical assets dispersed throughout a geographic region (e.g., abuilding, a town, a county, a state, a country). Each of the physicalassets includes hardware and software to perform its respectivefunctions within the system. A physical asset is a computing entity(CE), a public or provide networking device (ND), a user access device(UAD), or a business associate access device (BAAD).

A computing entity may be a user device, a system admin device, aserver, a printer, a data storage device, etc. A network device may be alocal area network device, a network card, a wide area network device,etc. A user access device is a portal that allows authorizes users ofthe system to remotely access the system. A business associated accessdevice is a portal that allows authorized business associates of thesystem access the system.

Some of the computing entities are grouped via a common connection to anetwork device, which provides the group of computing entities access toother parts of the system and/or the internet. For example, thehighlighted computing entity may access a publicly available server 25via network devices coupled to the network infrastructure. The analysissystem 10 can evaluation whether this is an appropriate access, theunderstanding of this access, the implementation to enable this access,and/or the operation of the system to support this access.

FIG. 9 is a schematic block diagram of an example of a system section ofa system selected for evaluation similar to FIG. 8. In this example,only a portion of the system is being tested, i.e., system section undertest 91. As such, the analysis system 10 only evaluates assets, systemfunctions, and/or security functions related to assets within the systemsection under test 91.

FIG. 10 is a schematic block diagram of another example of a systemsection of a system selected for evaluation similar to FIG. 9. In thisexample, a single computing entity (CE) is being tested, i.e., systemsection under test 91. As such, the analysis system 10 only evaluatesassets, system functions, and/or security functions related to theselected computing entity.

FIG. 11 is a schematic block diagram of an embodiment of a networkedenvironment having a system 11 (or system 12 or system 13), the analysissystem 10, one or more networks 14, one or more system proficiencyresources 22, one or more business associated computing devices 23, oneor more publicly available servers 27, one or more subscription basedservers 28, one or more BOT computing devices 25, and one or more badactor computing devices 26.

In this embodiment, the system 11 is shown to include a plurality ofsystem assets (SA). A system asset (SA) may include one or more systemsub assets (S2A) and a system sub asset (S2A) may include one or moresystem sub-sub assets (S3A). While being a part of the analysis system10, at least one data extraction module (DEM) 80 and at least one systemuser interface module (SUIM) 81 are installed on the system 11.

A system element includes one or more system assets. A system asset (SA)may be a physical asset or a conceptual asset as previously described.As an example, a system element includes a system asset of a computingdevice. The computing device, which is the SA, includes userapplications and an operating system; each of which are sub assets ofthe computing device (S2A). In addition, the computing device includes anetwork card, memory devices, etc., which are sub assets of thecomputing device (S2A). Documents created from a word processing userapplication are sub assets of the word processing user application (S3A)and sub-sub assets of the computing device.

As another example, the system asset (SA) includes a plurality ofcomputing devices, printers, servers, etc. of a department of theorganization operating the system 11. In this example, a computingdevice is a sub asset of the system asset and the software and hardwareof the computing devices are sub-sub assets.

The analysis system 10 may evaluate understanding, implementation,and/or operation of one or more system assets, one or more system subassets, and/or one or more system sub-sub assets, as an asset, as itsupports system functions 82, and/or as it supports security functions.The evaluation may be to produce an evaluation rating, to identifydeficiencies, and/or to auto-correct deficiencies.

FIG. 12 is a schematic block diagram of an embodiment of a system 11that includes a plurality of physical assets 100 coupled to an analysissystem 100. The physical assets 100 include an analysis interface device101, one or more networking devices 102, one or more security devices103, one or more system admin devices 104, one or more user devices 105,one or more storage devices 106, and/or one or more servers 107. Eachdevice may be a computing entity that includes hardware (HW) componentsand software (SW) applications (user, device, drivers, and/or system). Adevice may further include a data extraction module (DEM).

The analysis interface device 101 includes a data extraction module(DEM) 80 and the system user interface module 81 to provide connectivityto the analysis system 10. With the connectivity, the analysis system 10is able to evaluate understanding, implementation, and/or operation ofeach device, or portion thereof, as an asset, as it supports systemfunctions 82, and/or as it supports security functions. For example, theanalysis system 10 evaluates the understanding of networking devices 102as an asset. As a more specific example, the analysis system 10evaluates how well the networking devices 102, its hardware, and itssoftware are understood within the system and/or by the systemadministrators. The evaluation includes how well are the networkingdevices 102, its hardware, and its software documented; how well arethey implemented based on system requirements; how well do they operatebased on design and/or system requirements; how well are they maintainedper system policies and/or procedures; how well are their deficienciesidentified; and/or how well are their deficiencies auto-corrected.

FIG. 13 is a schematic block diagram of another embodiment of anetworked environment having a system 11 that includes a plurality ofsystem assets coupled to an analysis system 10. This embodiment issimilar to the embodiment of FIG. 11 with the addition of additionaldata extraction modules (DEM) 80. In this embodiment, each system asset(SA) is affiliated with its own DEM 80. This allows the analysis system10 to extract data more efficiently than via a single DEM. A furtherextension of this embodiment is that each system sub asset (S2A) couldhave its own DEM 80. As yet a further extension, each system sub-subasset (S3A) could have its own DEM 80.

FIG. 14 is a schematic block diagram of another embodiment of a system11 physical assets 100 coupled to an analysis system 100. The physicalassets 100 include one or more networking devices 102, one or moresecurity devices 103, one or more system admin devices 104, one or moreuser devices 105, one or more storage devices 106, and/or one or moreservers 107. Each device may be a computing entity that includeshardware (HW) components and software (SW) applications (user, system,and/or device).

The system admin device 104 includes one or more analysis system modules17, which includes a data extraction module (DEM) 80 and the system userinterface module 81 to provide connectivity to the analysis system 10.With the connectivity, the analysis system 10 is able to evaluateunderstanding, implementation, and/or operation of each device, orportion thereof, as an asset, as it supports system functions 82, and/oras it supports security functions. For example, the analysis system 10evaluates the implementation of networking devices 102 to support systemfunctions. As a more specific example, the analysis system 10 evaluateshow well the networking devices 102, its hardware, and its software areimplemented within the system to support one or more system functions(e.g., managing network traffic, controlling network access per businessguidelines, policies, and/or processes, etc.). The evaluation includeshow well is the implementation of the networking devices 102, itshardware, and its software documented to support the one or more systemfunctions; how well does their implementation support the one or moresystem functions; how well have their implementation to support the oneor more system functions been verified in accordance with policies,processes, etc.; how well are they updated per system policies and/orprocedures; how well are their deficiencies in support of the one ormore system functions identified; and/or how well are their deficienciesin support of the one or more system functions auto-corrected.

FIG. 15 is a schematic block diagram of another embodiment of a system11 that includes a plurality of physical assets 100 coupled to ananalysis system 100. The physical assets 100 include an analysisinterface device 101, one or more networking devices 102, one or moresecurity devices 103, one or more system admin devices 104, one or moreuser devices 105, one or more storage devices 106, and/or one or moreservers 107. Each device may be a computing entity that includeshardware (HW) components and software (SW) applications (user, device,drivers, and/or system). This embodiment is similar to the embodiment ofFIG. 12 with a difference being that the devices 102-107 do not includea data extraction module (DEM) as is shown in FIG. 12.

FIG. 16 is a schematic block diagram of another embodiment of a system11 that includes networking devices 102, security devices 103, servers107, storage devices 106, and user devices 105. The system 11 is coupledto the network 14, which provides connectivity to the business associatecomputing device 23. The network 14 is shown to include one or more widearea networks (WAN) 162, one or more wireless LAN (WLAN) and/or LANs164, one or more virtual private networks 166.

The networking devices 102 includes one or more modems 120, one or morerouters 121, one or more switches 122, one or more access points 124,and/or one or more local area network cards 124. The analysis system 10can evaluate the network devices 102 collectively as assets, as theysupport system functions, and/or as they support security functions. Theanalysis system 10 may also evaluate each network device individually asan asset, as it supports system functions, and/or as it supportssecurity functions. The analysis system may further evaluate one or morenetwork devices as part of the physical assets of a system aspect (e.g.,the system or a portion thereof being evaluated with respect to one ormore system criteria and one or more system modes).

The security devices 103 includes one or more infrastructure managementtools 125, one or more encryption software programs 126, one or moreidentity and access management tools 127, one or more data protectionsoftware programs 128, one or more system monitoring tools 129, one ormore exploit and malware protection tools 130, one or more vulnerabilitymanagement tools 131, and/or one or more data segmentation and boundarytools 132. Note that a tool is a program that functions to develop,repair, and/or enhance other programs and/or hardware.

The analysis system 10 can evaluate the security devices 103collectively as assets, as they support system functions, and/or as theysupport security functions. The analysis system 10 may also evaluateeach security device individually as an asset, as it supports systemfunctions, and/or as it supports security functions. The analysis systemmay further evaluate one or more security devices as part of thephysical assets of a system aspect (e.g., the system or a portionthereof being evaluated with respect to one or more system criteria andone or more system modes).

The servers 107 include one or more telephony servers 133, one or moreecommerce servers 134, one or more email servers 135, one or more webservers 136, and/or one or more content servers 137. The analysis system10 can evaluate the servers 103 collectively as assets, as they supportsystem functions, and/or as they support security functions. Theanalysis system 10 may also evaluate each server individually as anasset, as it supports system functions, and/or as it supports securityfunctions. The analysis system may further evaluate one or more serversas part of the physical assets of a system aspect (e.g., the system or aportion thereof being evaluated with respect to one or more systemcriteria and one or more system modes).

The storage devices includes one or more cloud storage devices 138, oneor more storage racks 139 (e.g., a plurality of storage devices mountedin a rack), and/or one or more databases 140. The analysis system 10 canevaluate the storage devices 103 collectively as assets, as they supportsystem functions, and/or as they support security functions. Theanalysis system 10 may also evaluate each storage device individually asan asset, as it supports system functions, and/or as it supportssecurity functions. The analysis system may further evaluate one or morestorage devices as part of the physical assets of a system aspect (e.g.,the system or a portion thereof being evaluated with respect to one ormore system criteria and one or more system modes).

The user devices 105 include one or more landline phones 141, one ormore IP cameras 144, one or more cell phones 143, one or more usercomputing devices 145, one or more IP phones 150, one or more videoconferencing equipment 148, one or more scanners 151, and/or one or moreprinters 142. The analysis system 10 can evaluate the use devices 103collectively as assets, as they support system functions, and/or as theysupport security functions. The analysis system 10 may also evaluateeach user device individually as an asset, as it supports systemfunctions, and/or as it supports security functions. The analysis systemmay further evaluate one or more user devices as part of the physicalassets of a system aspect (e.g., the system or a portion thereof beingevaluated with respect to one or more system criteria and one or moresystem modes).

The system admin devices 104 includes one or more system admin computingdevices 146, one or more system computing devices 194 (e.g., datamanagement, access control, privileges, etc.), and/or one or moresecurity management computing devices 147. The analysis system 10 canevaluate the system admin devices 103 collectively as assets, as theysupport system functions, and/or as they support security functions. Theanalysis system 10 may also evaluate each system admin deviceindividually as an asset, as it supports system functions, and/or as itsupports security functions. The analysis system may further evaluateone or more system admin devices as part of the physical assets of asystem aspect (e.g., the system or a portion thereof being evaluatedwith respect to one or more system criteria and one or more systemmodes).

FIG. 17 is a schematic block diagram of an embodiment of a usercomputing device 105 that includes software 160, a user interface 161,processing resources 163, memory 162 and one or more networking device164. The processing resources 163 include one or more processingmodules, cache memory, and a video graphics processing module.

The memory 162 includes non-volatile memory, volatile memory and/or diskmemory. The non-volatile memory stores hardware IDs, user credentials,security data, user IDs, passwords, access rights data, device IDs, oneor more IP addresses and security software. The volatile memory includessystem volatile memory and user volatile memory. The disk memoryincludes system disk memory and user disk memory. User memory (volatileand/or disk) stores user data and user applications. System memory(volatile and/or disk) stores system applications and system data.

The user interface 104 includes one or more I/O (input/output) devicessuch as video displays, keyboards, mice, eye scanners, microphones,speakers, and other devices that interface with one or more users. Theuser interface 161 further includes one or more physical (PHY) interfacewith supporting software such that the user computing device caninterface with peripheral devices.

The software 160 includes one or more I/O software interfaces (e.g.,drivers) that enable the processing module to interface with othercomponents. The software 160 also includes system applications, userapplications, disk memory software interfaces (drivers) and networksoftware interfaces (drivers).

The networking device 164 may be a network card or network interfacethat intercouples the user computing device 105 to devices external tothe computing device 105 and includes one or more PHY interfaces. Forexample, the network card is a WLAN card. As another example, thenetwork card is a cellular data network card. As yet another example,the network card is an ethernet card.

The user computing device may further include a data extraction module80. This would allow the analysis system 10 to obtain data directly fromthe user computing device. Regardless of how the analysis system 10obtains data regarding the user computing device, the analysis system 10can evaluate the user computing device as an asset, as it supports oneor more system functions, and/or as it supports one or more securityfunctions. The analysis system 10 may also evaluate each element of theuser computing device (e.g., each software application, each drive, eachpiece of hardware, etc.) individually as an asset, as it supports one ormore system functions, and/or as it supports one or more securityfunctions.

FIG. 18 is a schematic block diagram of an embodiment of a server 107that includes software 170, processing resources 171, memory 172 and oneor more networking resources 173. The processing resources 171 includeone or more processing modules, cache memory, and a video graphicsprocessing module. The memory 172 includes non-volatile memory, volatilememory, and/or disk memory. The non-volatile memory stores hardware IDs,user credentials, security data, user IDs, passwords, access rightsdata, device IDs, one or more IP addresses and security software. Thevolatile memory includes system volatile memory and shared volatilememory. The disk memory include server disk memory and shared diskmemory.

The software 170 includes one or more I/O software interfaces (e.g.,drivers) that enable the software 170 to interface with othercomponents. The software 170 includes system applications, serverapplications, disk memory software interfaces (drivers), and networksoftware interfaces (drivers). The networking resources 173 may be oneor more network cards that provides a physical interface for the serverto a network.

The server 107 may further include a data extraction module 80. Thiswould allow the analysis system 10 to obtain data directly from theserver. Regardless of how the analysis system 10 obtains data regardingthe server, the analysis system 10 can evaluate the server as an asset,as it supports one or more system functions, and/or as it supports oneor more security functions. The analysis system 10 may also evaluateeach element of the server (e.g., each software application, each drive,each piece of hardware, etc.) individually as an asset, as it supportsone or more system functions, and/or as it supports one or more securityfunctions.

FIG. 19 is a schematic block diagram of another embodiment of anetworked environment having a system 11 (or system 12 or system 13),the analysis system 10, one or more networks 14, one or more systemproficiency resources 22, one or more business associated computingdevices 23, one or more publicly available servers 27, one or moresubscription based servers 28, one or more BOT computing devices 25, andone or more bad actor computing devices 26.

In this embodiment, the system 11 is shown to include a plurality ofsystem functions (SF). A system function (SF) may include one or moresystem sub functions (S2F) and a system sub function (S2F) may includeone or more system sub-sub functions (S3F). While being a part of theanalysis system 10, at least one data extraction module (DEM) 80 and atleast one system user interface module (SUIM) 81 are installed on thesystem 11.

A system function (SF) includes one or more business operations, one ormore compliance requirements, one or more data flow objectives, one ormore data access control objectives, one or more data integrityobjectives, one or more data storage objectives, one or more data useobjectives, and/or one or more data dissemination objectives. Businessoperation system functions are the primary purpose for the system 11.The system 11 is designed and built to support the operations of thebusiness, which vary from business to business.

In general, business operations include operations regarding criticalbusiness functions, support functions for core business, product and/orservice functions, risk management objectives, business ecosystemobjectives, and/or business contingency plans. The business operationsmay be divided into executive management operations, informationtechnology operations, marketing operations, engineering operations,manufacturing operations, sales operations, accounting operations, humanresource operations, legal operations, intellectual property operations,and/or finance operations. Each type of business operation includessub-business operations, which, in turn may include its ownsub-operations.

For example, engineering operations includes a system function ofdesigning new products and/or product features. The design of a newproduct or feature involves sub-functions of creating designspecifications, creating a design based on the design specification, andtesting the design through simulation and/or prototyping. Each of thesesteps includes sub-steps. For example, for the design of a softwareprogram, the design process includes the sub-sub system functions ofcreating a high level design from the design specifications; creating alow level design from the high level design; and the creating code fromthe low level design.

A compliance requirement may be a regulatory compliance requirement, astandard compliance requirement, a statutory compliance requirement,and/or an organization compliance requirement. For example, there are aregulatory compliance requirements when the organization hasgovernmental agencies as clients. An example of a standard compliancerequirement, encryption protocols are often standardized. DataEncryption Standard (DES), Advanced Encryption Standard (AES), RSA(Rivest-Shamir-Adleman) encryption, and public-key infrastructure (PKI)are examples of encryption type standards. HIPAA (health InsurancePortability and Accountability Act) is an example of a statutorycompliance requirement. Examples of organization compliance requirementsinclude use of specific vendor hardware, use of specific vendorsoftware, use of encryption, etc.

A data flow objective is regarding where data can flow, at what ratedata can and should flow, the manner in which the data flow, and/or themeans over which the data flows. As an example of a data flow objective,data for remote storage is to flow via a secure data pipeline using aparticular encryption protocol. As another example of a data flowobjective, ingesting of data should have the capacity to handle a datarate of 100 giga-bits per second.

A data access control objective established which type of personneland/or type of assets can access specific types of data. For example,certain members of the corporate department and human resourcesdepartment have access to employee personnel files, while all othermembers of the organization do not.

A data integrity objective establishes a reliability that, when data isretrieved, it is the data that was stored, i.e., it was not lost,damaged, or corrupted. An example of a data integrity protocol is CyclicRedundancy Check (CRC). Another example of a data integrity protocol isa hash function.

A data storage objective establishes the manner in which data is to bestored. For example, a data storage objective is to store data in a RAIDsystem; in particular, a RAID 6 system. As another example, a datastorage objective is regarding archiving of data and the type of storageto use for archived data.

A data use objective establishes the manner in which data can be used.For example, if the data is for sale, then the data use objective wouldestablish what type of data is for sale, at what price, and what is thetarget customer. As another example, a data use objective establishesread only privileges, editing privileges, creation privileges, and/ordeleting privileges.

A data dissemination objective establishes how the data can be shared.For example, a data dissemination objective is regarding confidentialinformation and indicates how the confidential information should bemarked, who in can be shared with internally, and how it can be sharedexternally, if at all.

The analysis system 10 may evaluate understanding, implementation,and/or operation of one or more system functions, one or more system subfunctions, and/or one or more system sub-sub functions. The evaluationmay be to produce an evaluation rating, to identify deficiencies, and/orto auto-correct deficiencies. For example, the analysis system 10evaluates the understanding of the software development policies and/orprocesses. As another example, the analysis system 10 evaluates the useof software development policies and/or processes to implement asoftware program. As yet another example, analysis system 10 evaluatesthe operation of the software program with respect to the businessoperation, the design specifications, and/or the design.

FIG. 20 is a schematic block diagram of another embodiment of a system11 that includes, from a business operations perspective, divisions181-183, departments, and groups. The business structure of the system11, as in most businesses, is governed by a corporate department 180.The corporate department may have its own sub-system with structures andsoftware tailored to the corporate function of the system. Organizedunder the corporate department 180 are divisions, division 1 181,division 2 182, through division k 183. These divisions may be differentbusiness divisions of a multi-national conglomerate, may be differentfunctional divisions of a business, e.g., finance, marketing, sales,legal, engineering, research and development, etc. Under each division1081-183 include a plurality of departments. Under each department are anumber of groups.

The business structure is generic and can be used to represent thestructure of most conventional businesses and/or organizations. Theanalysis system 10 is able to use this generic structure to create andcategorize the business structure of the system 11. The creation andcategorization of the business structure is done in a number of ways.Firstly, the analysis system 10 accesses corporate organizationdocuments for the business and receive feedback from one or more personsin the business and use these documents and data to initially determineat least partially the business structure. Secondly, the analysis system10 determines the network structure of the other system, investigateidentities of components of the network structure, and construct asub-division of the other system. Then, based upon software used withinthe sub-division, data character, and usage character, the analysissystem 10 identifies more specifically the function of the divisions,departments and groups. In doing so, the analysis system 10 usesinformation known of third-party systems to assist in the analysis.

With the abstraction of the business structure, differing portions ofthe business structure may have different levels of abstraction from acomponent/sub-component/sub-sub-component/system/sub-system/sub-sub-systemlevel based upon characters of differing segments of the business. Forexample. a more detailed level of abstraction for elements of thecorporate and security departments of the business may be taken than forother departments of the business.

FIG. 21 is a schematic block diagram of another embodiment of a businessstructure of the system 11. Shown are a corporate department 180, an ITdepartment 181, division 2 182 through division “k” 183, where k is aninteger equal to or greater than 3. The corporate department 180includes a plurality of hardware devices 260, a plurality of softwareapplications 262, a plurality of business policies 264, a plurality ofbusiness procedures 266, local networking 268, a plurality of securitypolicies 270, a plurality of security procedures 272, data protectionresources 272, data access resources 276, data storage devices 278, apersonnel hierarchy 280, and external networking 282. Based upon anassessment of these assets of the corporate department 180, analysissystem 10 may evaluate the understanding, implementation, and/oroperation of the assets, system functions, and/or security functions ofthe corporate department from a number of different perspectives, aswill be described further with reference to one or more the subsequentfigures.

Likewise, the IT department 181 includes a plurality of hardware devices290, a plurality of software applications 292, a plurality of businesspolicies 294, a plurality of business procedures 296, local networking298, a plurality of security policies 300, a plurality of securityprocedures 302, data protection resources 304, data access resources306, data storage devices 308, a personnel hierarchy 310, and externalnetworking 312. Based upon an assessment of these assets of the ITdepartment 181, the analysis system 10 may evaluate the understanding,implementation, and/or operation of the assets, system functions, and/orsecurity functions of the IT department from a number of differentperspectives, as will be described further with reference to one or moreof the subsequent figures.

FIG. 22 is a schematic block diagram of another embodiment of a division182 of a system that includes multiple departments. The departmentsinclude a marketing department 190, an operations department 191, anengineering department 192, a manufacturing department 193, a salesdepartment 194, and an accounting department 195. Each of thedepartments includes a plurality of components relevant to support thecorresponding business functions and/or security functions of thedivision and of the department. In particular, the marketing department190 includes a plurality of devices, software, security policies,security procedures, business policies, business procedures, dataprotection resources, data access resources, data storage resources, apersonnel hierarchy, local network resources, and external networkresources.

Likewise, each of the operations department 191, the engineeringdepartment 192, the manufacturing department 193, the sales department194, and the accounting department 195 includes a plurality of devices,software, security policies, security procedures, business policies,business procedures, data protection resources, data access resources,data storage resources, a personnel hierarchy, local network resources,and external network resources.

Further, within the business structure, a service mesh may beestablished to more effectively protect important portions of thebusiness from other portions of the business. The service mesh may havemore restrictive safety and security mechanisms for one part of thebusiness than another portion of the business, e.g., manufacturingdepartment service mesh is more restrictive than the sales departmentservice mesh.

The analysis system 10 may evaluate the understanding, implementation,and/or operation of the assets, system functions, and/or securityfunctions of the division 182, of each department, of each type ofsystem elements, and/or each system element. For example, the analysissystem 10 evaluates the data access policies and procedures of eachdepartment. As another example, the analysis system 10 evaluates thedata storage policies, procedures, design, implementation, and/oroperation of data storage within the engineering department 192.

FIG. 23 is a schematic block diagram of another embodiment of anetworked environment having a system 11 (or system 12 or system 13),the analysis system 10, one or more networks 14, one or more systemproficiency resources 22, one or more business associated computingdevices 23, one or more publicly available servers 27, one or moresubscription based servers 28, one or more BOT computing devices 25, andone or more bad actor computing devices 26.

In this embodiment, the system 11 is shown to include a plurality ofsecurity functions (SEF). A security function (SEF) may include one ormore system sub security functions (SE2F) and a security sub function(SE2F) may include one or more security sub-sub functions (SE3F). Whilebeing a part of the analysis system 10, at least one data extractionmodule (DEM) 80 and at least one system user interface module (SUIM) 81are installed on the system 11. As used herein, a security functionincludes a security operation, a security requirement, a securitypolicy, and/or a security objective with respect to data, system access,system design, system operation, and/or system modifications (e.g.,updates, expansion, part replacement, maintenance, etc.).

A security function (SF) includes one or more threat detectionfunctions, one or more threat avoidance functions, one or more threatresolution functions, one or more threat recovery functions, one or morethreat assessment functions, one or more threat impact functions, one ormore threat tolerance functions, one or more business securityfunctions, one or more governance security functions, one or more dataat rest protection functions, one or more data in transit protectionfunctions, and/or one or more data loss prevention functions.

A threat detection function includes detecting unauthorized systemaccess; detecting unauthorized data access; detecting unauthorized datachanges; detecting uploading of worms, viruses, and the like; and/ordetecting bad actor attacks. A threat avoidance function includesavoiding unauthorized system access; avoiding unauthorized data access;avoiding unauthorized data changes; avoiding uploading of worms,viruses, and the like; and/or avoiding bad actor attacks.

A threat resolution function includes resolving unauthorized systemaccess; resolving unauthorized data access; resolving unauthorized datachanges; resolving uploading of worms, viruses, and the like; and/orresolving bad actor attacks. A threat recovery function includesrecovering from an unauthorized system access; recovering from anunauthorized data access; recovering from an unauthorized data changes;recovering from an uploading of worms, viruses, and the like; and/orrecovering from a bad actor attack.

A threat assessment function includes accessing the likelihood of and/ormechanisms for unauthorized system access; accessing the likelihood ofand/or mechanisms for unauthorized data access; accessing the likelihoodof and/or mechanisms for unauthorized data changes; accessing thelikelihood of and/or mechanisms for uploading of worms, viruses, and thelike; and/or accessing the likelihood of and/or mechanisms for bad actorattacks.

A threat impact function includes determining an impact on businessoperations from an unauthorized system access; resolving unauthorizeddata access; determining an impact on business operations from anunauthorized data changes; determining an impact on business operationsfrom an uploading of worms, viruses, and the like; and/or determining animpact on business operations from an bad actor attacks.

A threat tolerance function includes determining a level of tolerancefor an unauthorized system access; determining a level of tolerance foran unauthorized data access; determining a level of tolerance for anunauthorized data changes; determining a level of tolerance for anuploading of worms, viruses, and the like; and/or determining a level oftolerance for an bad actor attacks.

A business security function includes data encryption, handling of thirdparty data, releasing data to the public, and so on. A governancesecurity function includes HIPAA compliance; data creation, data use,data storage, and/or data dissemination for specific types of customers(e.g., governmental agency); and/or the like.

A data at rest protection function includes a data access protocol(e.g., user ID, password, etc.) to store data in and/or retrieve datafrom system data storage; data storage requirements, which include typeof storage, location of storage, and storage capacity; and/or other datastorage security functions.

A data in transit protection function includes using a specific datatransportation protocol (e.g., TCP/IP); using an encryption functionprior to data transmission; using an error encoding function for datatransmission; using a specified data communication path for datatransmission; and/or other means to protect data in transit. A data lossprevention function includes a storage encoding technique (e.g., singleparity encoding, double parity encoding, erasure encoding, etc.); astorage backup technique (e.g., one or two backup copies, erasureencoding, etc.); hardware maintenance and replacement policies andprocesses; and/or other means to prevent loss of data.

The analysis system 10 may evaluate understanding, implementation,and/or operation of one or more security functions, one or more securitysub functions, and/or one or more security sub-sub functions. Theevaluation may be to produce an evaluation rating, to identifydeficiencies, and/or to auto-correct deficiencies. For example, theanalysis system 10 evaluates the understanding of the threat detectionpolicies and/or processes. As another example, the analysis system 10evaluates the use of threat detection policies and/or processes toimplement a security assets. As yet another example, analysis system 10evaluates the operation of the security assets with respect to thethreat detection operation, the threat detection design specifications,and/or the threat detection design.

FIG. 24 is a schematic block diagram of an embodiment of an engineeringdepartment 200 of a division 182 that reports to a corporate department180 of a system 11. The engineering department 200 includes engineeringassets, engineering system functions, and engineering securityfunctions. The engineering assets include security HW & SW, user deviceHW & SW, networking HW & SW, system HW & SW, system monitoring HW & SW,and/or other devices that includes HW and/or SW.

In this example, the organization's system functions includes businessoperations, compliance requirements, data flow objectives, data accessobjectives, data integrity objectives, data storage objectives, data useobjectives, and/or data dissemination objectives. These system functionsapply throughout the system including throughout division 2 and for theengineering department 200 of division 2.

The division 182, however, can issues more restrictive, more secure,and/or more detailed system functions. In this example, the division hasissued more restrictive, secure, and/or detailed business operations(business operations +) and more restrictive, secure, and/or detaileddata access functions (data access +). Similarly, the engineeringdepartment 200 may issue more restrictive, more secure, and/or moredetailed system functions than the organization and/or the division. Inthis example, the engineering department has issued more restrictive,secure, and/or detailed business operations (business operations ++)than the division; has issued more restrictive, secure, and/or detaileddata flow functions (data flow ++) than the organization; has issuedmore restrictive, secure, and/or detailed data integrity functions (dataintegrity ++) than the organization; and has issued more restrictive,secure, and/or detailed data storage functions (data storage ++) thanthe organization.

For example, an organization level business operation regarding thedesign of new products and/or of new product features specifieshigh-level design and verify guidelines. The division issued moredetailed design and verify guidelines. The engineering department issuedeven more detailed design and verify guidelines.

The analysis system 10 can evaluate the compliance with the systemfunctions for the various levels. In addition, the analysis system 10can evaluate that the division issued system functions are compliantwith the organization issued system functions and/or are morerestrictive, more secure, and/or more detailed. Similarly, the analysissystem 10 can evaluate that the engineering department issued systemfunctions are compliant with the organization and the division issuedsystem functions and/or are more restrictive, more secure, and/or moredetailed.

As is further shown in this example, the organization security functionsincludes data at rest protection, data loss prevention, data in transitprotection, threat management, security governance, and businesssecurity. The division has issued more restrictive, more secure, and/ormore detailed business security functions (business security +). Theengineering department has issued more restrictive, more secure, and/ormore detailed data at rest protection (data at rest protection ++), dataloss prevention (data loss prevention ++), and data in transitprotection (data in transit ++).

The analysis system 10 can evaluate the compliance with the securityfunctions for the various levels. In addition, the analysis system 10can evaluate that the division issued security functions are compliantwith the organization issued security functions and/or are morerestrictive, more secure, and/or more detailed. Similarly, the analysissystem 10 can evaluate that the engineering department issued securityfunctions are compliant with the organization and the division issuedsecurity functions and/or are more restrictive, more secure, and/or moredetailed.

FIG. 25 is a schematic block diagram of an example of an analysis system10 evaluating a system element under test 91 of a system 11. The systemelement under test 91 corresponds to a system aspect (or system sector),which includes one or more system elements, one or more system criteria,and one or more system modes.

In this example, the system criteria are shown to includes guidelines,system requirements, system design & system build (systemimplementation), and the resulting system. The analysis system 10 mayevaluate the system, or portion thereof, during initial systemrequirement development, initial design of the system, initial build ofthe system, operation of the initial system, revisions to the systemrequirements, revisions to the system design, revisions to the systembuild, and/or operation of the revised system. A revision to a systemincludes adding assets, system functions, and/or security functions;deleting assets, system functions, and/or security functions; and/ormodifying assets, system functions, and/or security functions.

The guidelines include one or more of business objectives, securityobjectives, NIST cybersecurity guidelines, system objectives,governmental and/or regulatory requirements, third party requirements,etc. and are used to help create the system requirements. Systemrequirements outline the hardware requirements for the system, thesoftware requirements for the system, the networking requirements forthe system, the security requirements for the system, the logical dataflow for the system, the hardware architecture for the system, thesoftware architecture for the system, the logical inputs and outputs ofthe system, the system input requirements, the system outputrequirements, the system's storage requirements, the processingrequirements for the system, system controls, system backup, data accessparameters, and/or specification for other system features.

The system requirements are used to help create the system design. Thesystem design includes a high level design (HDL), a low level design(LLD), a detailed level design (DLD), and/or other design levels. Highlevel design is a general design of the system. It includes adescription of system architecture; a database design; an outline ofplatforms, services, and processes the system will require; adescription of relationships between the assets, system functions, andsecurity functions; diagrams regarding data flow; flowcharts; datastructures; and/or other documentation to enable more detailed design ofthe system.

Low level design is a component level design that is based on the HLD.It provides the details and definitions for every system component(e.g., HW and SW). In particular, LLD specifies the features of thesystem components and component specifications. Detailed level designdescribes the interaction of every component of the system.

The system is built based on the design to produce a resulting system(i.e., the implemented assets). The assets of system operate to performthe system functions and/or security functions.

The analysis system 10 can evaluate the understanding, implementation,operation and/or self-analysis of the system 11 at one or more systemcriteria level (e.g., guidelines, system requirements, systemimplementation (e.g., design and/or build), and system) in a variety ofways.

The analysis system 10 evaluates the understanding of the system (orportion thereof) by determining a knowledge level of the system and/ormaturity level of system. For example, an understanding evaluationinterprets what is known about the system and compares it to what shouldbe known about the system.

As a more specific example, the analysis system evaluates theunderstanding of the guidelines. For instance, the analysis system 10evaluates the policies, processes, procedures, automation,certifications, documentation, and/or other evaluation metric (e.g.,evaluation metrics) regarding the thoroughness of the guidelines tofacilitate the understanding of the guidelines. The more incomplete thedata regarding the evaluation metrics, the more likely the guidelinesare incomplete; which indicates a lack of understanding. The fewernumbers of and/or incompleteness of policies, processes, procedures,automation, documentation, certification, and/or other evaluation metricregarding the creation and/or use of the guidelines, the more likely theguidelines are not well understood (e.g., lower level of knowledgeand/or of system maturity) resulting in a low evaluation rating.

As another more specific example of an understanding evaluation, theanalysis system 10 evaluates the understanding of the systemrequirements. For instance, the analysis system 10 evaluates thepolicies, processes, procedures, automation, certifications,documentation, and/or other evaluation metric (e.g., evaluation metrics)regarding the thoroughness of the system requirements to facilitate theunderstanding of the system requirements. The more incomplete the dataregarding the evaluation metrics, the more likely the systemrequirements are incomplete; which indicates a lack of understanding.The fewer numbers of and/or incompleteness of policies, processes,procedures, automation, documentation, certification, and/or otherevaluation metric regarding the creation and/or use of the systemrequirements, the more likely the system requirements are not wellunderstood (e.g., lower level of knowledge and/or of system maturity)resulting in a low evaluation rating.

As another more specific example of an understanding evaluation, theanalysis system 10 evaluates the understanding of the system design. Forinstance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the thoroughnessof the system design to facilitate the understanding of the systemdesign. The more incomplete the data regarding the evaluation metrics,the more likely the system design is incomplete; which indicates a lackof understanding. The fewer numbers of and/or incompleteness ofpolicies, processes, procedures, automation, documentation,certification, and/or other evaluation metric regarding the creationand/or use of the system design, the more likely the system design isnot well understood (e.g., lower level of knowledge and/or of systemmaturity) resulting in a low evaluation rating.

As another more specific example of an understanding evaluation, theanalysis system 10 evaluates the understanding of the system build. Forinstance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the thoroughnessof the system build to facilitate the understanding of the system build.The more incomplete the data regarding the evaluation metrics, the morelikely the system build is incomplete; which indicates a lack ofunderstanding. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the execution of and/or use of thesystem build, the more likely the system build is not well understood(e.g., lower level of knowledge and/or of system maturity) resulting ina low evaluation rating.

As another more specific example of an understanding evaluation, theanalysis system 10 evaluates the understanding of the system functions.For instance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the thoroughnessof the system build to facilitate the understanding of the system build.The more incomplete the data regarding the evaluation metrics, the morelikely the system build is incomplete; which indicates a lack ofunderstanding. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the execution of and/or use of thesystem build, the more likely the system build is not well understood(e.g., lower level of knowledge and/or of system maturity) resulting ina low evaluation rating.

As another more specific example of an understanding evaluation, theanalysis system 10 evaluates the understanding of the securityfunctions. For instance, the analysis system 10 evaluates the policies,processes, procedures, automation, certifications, documentation, and/orother evaluation metric (e.g., evaluation metrics) regarding thethoroughness of the system functions to facilitate the understanding ofthe system functions. The more incomplete the data regarding theevaluation metrics, the more likely the system functions are incomplete;which indicates a lack of understanding. The fewer numbers of and/orincompleteness of policies, processes, procedures, automation,documentation, certification, and/or other evaluation metric regardingthe execution of and/or use of the system functions, the more likely thesystem functions are not well understood (e.g., lower level of knowledgeand/or of system maturity) resulting in a low evaluation rating.

As another more specific example of an understanding evaluation, theanalysis system 10 evaluates the understanding of the system assets. Forinstance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the thoroughnessof the system assets to facilitate the understanding of the systemassets. The more incomplete the data regarding the evaluation metrics,the more likely the system assets are incomplete; which indicates a lackof understanding. The fewer numbers of and/or incompleteness ofpolicies, processes, procedures, automation, documentation,certification, and/or other evaluation metric regarding the selection,identification, and/or use of the system assets, the more likely thesystem assets are not well understood (e.g., lower level of knowledgeand/or of system maturity) resulting in a low evaluation rating.

The analysis system 10 also evaluates the implementation of the system(or portion thereof) by determining how well the system is being, wasdeveloped, and/or is being updated. For example, the analysis system 10determines how well the assets, system functions, and/or securityfunctions are being developed, have been developed, and/or are beingupdated based on the guidelines, the system requirements, the systemdesign, and/or the system build.

As a more specific example of an implementation evaluation, the analysissystem 10 evaluates the implementation of the guidelines. For instance,the analysis system 10 evaluates the policies, processes, procedures,automation, certifications, documentation, and/or other evaluationmetric (e.g., evaluation metrics) regarding the development of theguidelines. The more incomplete the data regarding the evaluationmetrics, the more likely the development of the guidelines isincomplete. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the development of the guidelines, themore likely the guidelines are not well developed (e.g., lower level ofsystem development maturity) resulting in a low evaluation rating.

As another more specific example of an implementation evaluation, theanalysis system 10 evaluates the implementation of the systemrequirements. For instance, the analysis system 10 evaluates thepolicies, processes, procedures, automation, certifications,documentation, and/or other evaluation metric (e.g., evaluation metrics)regarding the development of the system requirements. The moreincomplete the data regarding the evaluation metrics, the more likelythe development of the system requirements is incomplete. The fewernumbers of and/or incompleteness of policies, processes, procedures,automation, documentation, certification, and/or other evaluation metricregarding the development of the system requirements, the more likelythe system requirements are not well developed (e.g., lower level ofsystem development maturity) resulting in a low evaluation rating.

As another more specific example of an implementation evaluation, theanalysis system 10 evaluates the implementation of the system design.For instance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the developmentof the system design. The more incomplete the data regarding theevaluation metrics, the more likely the development of the system designis incomplete. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the development of the system design,the more likely the system design is not well developed (e.g., lowerlevel of system development maturity) resulting in a low evaluationrating.

As another more specific example of an implementation evaluation, theanalysis system 10 evaluates the implementation of the system build. Forinstance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the developmentof the system build. The more incomplete the data regarding theevaluation metrics, the more likely the development of the system buildis incomplete. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the development of the system build,the more likely the system build is not well developed (e.g., lowerlevel of system development maturity) resulting in a low evaluationrating.

As another more specific example of an implementation evaluation, theanalysis system 10 evaluates the implementation of the system functions.For instance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the developmentof the system functions. The more incomplete the data regarding theevaluation metrics, the more likely the development of the systemfunctions is incomplete. The fewer numbers of and/or incompleteness ofpolicies, processes, procedures, automation, documentation,certification, and/or other evaluation metric regarding the developmentof the system functions, the more likely the system functions are notwell developed (e.g., lower level of system development maturity)resulting in a low evaluation rating.

As another more specific example of an implementation evaluation, theanalysis system 10 evaluates the implementation of the securityfunctions. For instance, the analysis system 10 evaluates the policies,processes, procedures, automation, certifications, documentation, and/orother evaluation metric (e.g., evaluation metrics) regarding thedevelopment of the security functions. The more incomplete the dataregarding the evaluation metrics, the more likely the development of thesecurity functions is incomplete. The fewer numbers of and/orincompleteness of policies, processes, procedures, automation,documentation, certification, and/or other evaluation metric regardingthe development of the security functions, the more likely the securityfunctions are not well developed (e.g., lower level of systemdevelopment maturity) resulting in a low evaluation rating.

As another more specific example of an implementation evaluation, theanalysis system 10 evaluates the implementation of the system assets.For instance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the developmentof the system assets. The more incomplete the data regarding theevaluation metrics, the more likely the development of the system assetsis incomplete. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the development of the system assets,the more likely the system assets are not well developed (e.g., lowerlevel of system development maturity) resulting in a low evaluationrating.

The analysis system 10 also evaluates the operation of the system (orportion thereof) by determining how well the system fulfills itsobjectives. For example, the analysis system 10 determines how well theassets, system functions, and/or security functions to fulfill theguidelines, the system requirements, the system design, the systembuild, the objectives of the system, and/or other purpose of the system.

As a more specific example of an operation evaluation, the analysissystem 10 evaluates the operation (i.e., fulfillment) of the guidelinesby the system requirements. For instance, the analysis system 10evaluates the policies, processes, procedures, automation,certifications, documentation, and/or other evaluation metric (e.g.,evaluation metrics) regarding the fulfillment of the guidelines by thesystem requirements. The more incomplete the data regarding theevaluation metrics, the more likely the fulfillment of the guidelines bythe system requirements is incomplete. The fewer numbers of and/orincompleteness of policies, processes, procedures, automation,documentation, certification, and/or other evaluation metric regardingthe fulfillment of the guidelines by the system requirements, the morelikely the system requirements does not adequately fulfill theguidelines (e.g., lower level of system development maturity) resultingin a low evaluation rating.

As another more specific example of an operation evaluation, theanalysis system 10 evaluates the operation (i.e., fulfillment) of theguidelines and/or the system requirements by the system design. Forinstance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the fulfillmentof the guidelines and/or the system requirements by the system design.The more incomplete the data regarding the evaluation metrics, the morelikely the fulfillment of the guidelines and/or the system requirementsby the system design is incomplete. The fewer numbers of and/orincompleteness of policies, processes, procedures, automation,documentation, certification, and/or other evaluation metric regardingthe fulfillment of the guidelines and/or the system requirements by thesystem design, the more likely the system design does not adequatelyfulfill the guidelines and/or the system requirements (e.g., lower levelof system operation maturity) resulting in a low evaluation rating.

As another more specific example of an operation evaluation, theanalysis system 10 evaluates the operation (i.e., fulfillment) of theguidelines, the system requirements, and/or the system design by thesystem build. For instance, the analysis system 10 evaluates thepolicies, processes, procedures, automation, certifications,documentation, and/or other evaluation metric (e.g., evaluation metrics)regarding the fulfillment of the guidelines, the system requirements,and/or the system design by the system build. The more incomplete thedata regarding the evaluation metrics, the more likely the fulfillmentof the guidelines, the system requirements, and/or the system design bythe system build is incomplete. The fewer numbers of and/orincompleteness of policies, processes, procedures, automation,documentation, certification, and/or other evaluation metric regardingthe fulfillment of the guidelines, the system requirements, and/or thesystem design by the system build, the more likely the system build doesnot adequately fulfill the guidelines, the system requirements, and/orthe system design (e.g., lower level of system operation maturity)resulting in a low evaluation rating.

As another more specific example of an operation evaluation, theanalysis system 10 evaluates the operation (i.e., fulfillment) of theguidelines, the system requirements, the system design, the systembuild, and/or objectives by the operation of the system in performingthe system functions. For instance, the analysis system 10 evaluates thepolicies, processes, procedures, automation, certifications,documentation, and/or other evaluation metric (e.g., evaluation metrics)regarding the fulfillment of the guidelines, the system requirements,the system design, the system build, and/or objectives regarding theperformance of the system functions by the system. The more incompletethe data regarding the evaluation metrics, the more likely thefulfillment of the guidelines, the system requirements, the systemdesign, the system, and/or the objectives regarding the system functionsis incomplete. The fewer numbers of and/or incompleteness of policies,processes, procedures, automation, documentation, certification, and/orother evaluation metric regarding the fulfillment of the guidelines, thesystem requirements, the system design, the system build, and/or theobjectives, the more likely the system does not adequately fulfill theguidelines, the system requirements, the system design, the systembuild, and/or the objectives regarding the system functions (e.g., lowerlevel of system operation maturity) resulting in a low evaluationrating.

As another more specific example of an operation evaluation, theanalysis system 10 evaluates the operation (i.e., fulfillment) of theguidelines, the system requirements, the system design, the systembuild, and/or objectives by the operation of the system in performingthe security functions. For instance, the analysis system 10 evaluatesthe policies, processes, procedures, automation, certifications,documentation, and/or other evaluation metric (e.g., evaluation metrics)regarding the fulfillment of the guidelines, the system requirements,the system design, the system build, and/or objectives regarding theperformance of the security functions by the system. The more incompletethe data regarding the evaluation metrics, the more likely thefulfillment of the guidelines, the system requirements, the systemdesign, the system, and/or the objectives regarding the securityfunctions is incomplete. The fewer numbers of and/or incompleteness ofpolicies, processes, procedures, automation, documentation,certification, and/or other evaluation metric regarding the fulfillmentof the guidelines, the system requirements, the system design, thesystem build, and/or the objectives, the more likely the system does notadequately fulfill the guidelines, the system requirements, the systemdesign, the system build, and/or the objectives regarding the securityfunctions (e.g., lower level of system operation maturity) resulting ina low evaluation rating.

As another more specific example of an operation evaluation, theanalysis system 10 evaluates the operation (i.e., fulfillment) of theguidelines, the system requirements, the system design, the systembuild, and/or objectives by the operation of the system functions. Forinstance, the analysis system 10 evaluates the policies, processes,procedures, automation, certifications, documentation, and/or otherevaluation metric (e.g., evaluation metrics) regarding the fulfillmentof the guidelines, the system requirements, the system design, thesystem build, and/or objectives regarding the performance of the systemassets. The more incomplete the data regarding the evaluation metrics,the more likely the fulfillment of the guidelines, the systemrequirements, the system design, the system, and/or the objectivesregarding the system assets is incomplete. The fewer numbers of and/orincompleteness of policies, processes, procedures, automation,documentation, certification, and/or other evaluation metric regardingthe fulfillment of the guidelines, the system requirements, the systemdesign, the system build, and/or the objectives, the more likely thesystem assets do not adequately fulfill the guidelines, the systemrequirements, the system design, the system build, and/or the objectives(e.g., lower level of system operation maturity) resulting in a lowevaluation rating.

The analysis system 10 also evaluates the self-analysis capabilities ofthe system (or portion thereof) by determining how well theself-analysis functions are implemented and how they subsequentlyfulfill the self-analysis objectives. In an example, the self-analysiscapabilities of the system are a self-analysis system that overlies thesystem. Accordingly, the overlaid self-analysis system can be evaluatedby the analysis system 10 in a similar manner as the system under test91. For example, the understanding, implementation, and/or operation ofthe overlaid self-analysis system can be evaluated with respect toself-analysis guidelines, self-analysis requirements, design of theself-analysis system, build of the self-analysis system, and/oroperation of the self-analysis system

As part of the evaluation process, the analysis system 10 may identifydeficiencies and, when appropriate, auto-correct a deficiency. Forexample, the analysis system 10 identifies deficiencies in theunderstanding, implementation, and/or operation of the guidelines, thesystem requirements, the system design, the system build, the resultingsystem, and/or the system objectives. For example, the analysis system10 obtains addition information from the system via a data gatheringprocess (e.g., producing discovered data) and/or from a systemproficiency resource (e.g., producing desired data). The analysis system10 uses the discovered data and/or desired data to identify thedeficiencies. When possible, the analysis system 10 auto-corrects thedeficiencies. For example, when a software tool that aides in thecreation of guidelines and/or system requirements is missing from thesystem's tool set, the analysis system 10 can automatically obtain acopy of the missing software tool for the system.

FIG. 26 is a schematic block diagram of another example of an analysissystem 10 evaluating a system element under test 91. In this example,the analysis system 10 is evaluating the system element under test 91from three evaluation viewpoints: disclosed data, discovered data, anddesired data. Disclosed data is the known data of the system at theoutset of an analysis, which is typically supplied by a systemadministrator and/or is obtained from data files of the system.Discovered data is the data discovered about the system from the by theanalysis system 10 during the analysis. Desired data is the dataobtained by the analysis system 10 from system proficiency resourcesregarding desired guidelines, system requirements, system design, systembuild, and/or system operation.

The evaluation from the three evaluation viewpoints may be doneserially, in parallel, and/or in a parallel-serial combination toproduce three sets of evaluation ratings. One set for disclosed data,one set for discovered data, and one set for desired data.

A set of evaluation ratings includes one or more of: an evaluationrating regarding the understanding of the guidelines; an evaluationrating regarding the understanding of the system requirements; anevaluation rating regarding the understanding of the system design; anevaluation rating regarding the understanding of the system build; anevaluation rating regarding the understanding of the system operation;an evaluation rating regarding the development of the systemrequirements from the guidelines; an evaluation rating regarding thedesign from the system requirements; an evaluation rating regarding thesystem build from the design; an evaluation rating regarding the systemoperation based on the system design and/or system build; an evaluationrating regarding the guidelines; an evaluation rating regarding thesystem requirements; an evaluation rating regarding the system design;an evaluation rating regarding the system build; and/or an evaluationrating regarding the system operation.

FIG. 27 is a schematic block diagram of another example of an analysissystem 10 evaluating a system element under test 91. In this example,the analysis system 10 is evaluating the system element under test 91from three evaluation viewpoints: disclosed data, discovered data, anddesired data with regard to security functions. The evaluation from thethree evaluation viewpoints for the security functions may be doneserially, in parallel, and/or in a parallel-serial combination toproduce three sets of evaluation ratings with respect to securityfunctions: one for disclosed data, one for discovered data, and one fordesired data.

FIG. 28 is a schematic block diagram of another example of an analysissystem 10 evaluating a system element under test 91. In this example,the analysis system 10 is evaluating the system element under test 91from three evaluation viewpoints and from three evaluation modes. Forexample, disclosed data regarding assets, discovered data regardingassets, desired data regarding assets, disclosed data regarding systemfunctions, discovered data regarding system functions, desired dataregarding system functions, disclosed data regarding security functions,discovered data regarding security functions, and desired data regardingsecurity functions.

The evaluation from the nine evaluation viewpoints & evaluation modecombinations may be done serially, in parallel, and/or in aparallel-serial combination to produce nine sets of evaluation ratingsone for disclosed data regarding assets, one for discovered dataregarding assets, one for desired data regarding assets, one fordisclosed data regarding system functions, one for discovered dataregarding system functions, one for desired data regarding functions,one for disclosed data regarding security functions, one for discovereddata regarding security functions, and one for desired data regardingsecurity functions.

FIG. 29 is a schematic block diagram of an example of the functioning ofan analysis system 10 evaluating a system element under test 91.Functionally, the analysis system 10 includes evaluation criteria 211,evaluation mode 212, analysis perspective 213, analysis viewpoint 214,analysis categories 215, data gathering 216, pre-processing 217, andanalysis metrics 218 to produce one or more ratings 219. The evaluationcriteria 211 includes guidelines, system requirements, system design,system build, and system operation. The evaluation mode 212 includesassets, system functions, and security functions. The evaluationcriteria 211 and the evaluation mode 212 are part of the system aspect,which corresponds to the system, or portion thereof, being evaluated.

The analysis perspective 213 includes understanding, implementation,operation, and self-analysis. The analysis viewpoint includes disclosed,discovered, and desired. The analysis categories 215 include identify,protect, detect, respond, and recover. The analysis perspective 213, theanalysis viewpoint 214, and the analysis categories correspond to howthe system, or portion thereof, will be evaluated. For example, thesystem, or portion thereof, is being evaluated regarding theunderstanding of the system's ability to identify assets, systemfunctions, and/or security functions from discovered data.

The analysis metrics 218 includes process, policy, procedure,automation, certification, and documentation. The analysis metric 218and the pre-processing 217 corresponds to manner of evaluation. Forexample, the policies regarding system's ability to identify assets,system functions, and/or security functions from discovered data of thesystem, or portion thereof, are evaluated to produce an understandingevaluation rating.

In an example of operation, the analysis system 10 determines whatportion of the system is evaluated (i.e., a system aspect). As such, theanalysis system 10 determines one or more system elements (e.g.,including one or more system assets which are one or more physicalassets and/or conceptual assets), one or more system criteria (e.g.,guidelines, system requirements, system design, system build, and/orsystem operation), and one or more system modes (e.g., assets, systemfunctions, and security functions). The analysis system 10 may determinethe system aspect in a variety of ways. For example, the analysis system10 receives an input identifying the system aspect from an authorizedoperator of the system (e.g., IT personnel, executive personnel, etc.).As another example, the analysis system determines the system aspect ina systematic manner to evaluate various combinations of system aspectsas part of an overall system evaluation. The overall system evaluationmay be done one time, periodically, or continuously. As yet anotherexample, the analysis system determines the system aspect as part of asystematic analysis of a section of the system, which may be done onetime, periodically, or continuously.

The analysis system then determines how the system aspect is to beevaluated by selecting one or more analysis perspectives (understanding,implementation, operation, and self-analysis), one or more analysisviewpoints (disclosed, discovered, and desired), and one or moreanalysis categories (identify, protect, detect, respond, and recover).The analysis system 10 may determine how the system aspect is to beevaluated in a variety of ways. For example, the analysis system 10receives an input identifying how the system aspect is to be evaluatedfrom an authorized operator of the system (e.g., IT personnel, executivepersonnel, etc.). As another example, the analysis system determines howthe system aspect is to be evaluated in a systematic manner to evaluatethe system aspect in various combinations of analysis perspectives,analysis viewpoints, and analysis categories as part of an overallsystem evaluation. The overall system evaluation may be done one time,periodically, or continuously. As yet another example, the analysissystem determines how the system aspect is to be evaluated as part of asystematic analysis of a section of the system, which may be done onetime, periodically, or continuously.

The analysis system 10 also determines one or more analysis metrics(e.g., process, policy, procedure, automation, certification, anddocumentation) regarding the manner for evaluating the system aspect inaccordance with how it's to be evaluated. A policy sets out a strategicdirection and includes high-level rules or contracts regarding issuesand/or matters. For example, all software shall be a most recent versionof the software. A process is a set of actions for generating outputsfrom inputs and includes one or more directives for generating outputsfrom inputs. For example, a process regarding the software policy isthat software updates are to be performed by the IT department and allsoftware shall be updated within one month of the release of the newversion of software.

A procedure is the working instructions to complete an action as may beoutlined by a process. For example, the IT department handling softwareupdates includes a procedure that describes the steps for updating thesoftware, verifying that the updated software works, and recording theupdating and verification in a software update log. Automation is inregard to the level of automation the system includes for handlingactions, issues, and/or matters of policies, processes, and/orprocedures. Documentation is in regard to the level of documentation thesystem has regard guidelines, system requirements, system design, systembuild, system operation, system assets, system functions, securityfunctions, system understanding, system implementation, operation of thesystem, policies, processes, procedures, etc. Certification is in regardto certifications of the system, such as maintenance certification,regulatory certifications, etc.

In an example, the analysis system 10 receives an input identifyingmanner in which to evaluate the system aspect from an authorizedoperator of the system (e.g., IT personnel, executive personnel, etc.).As another example, the analysis system determines the manner in whichto evaluate the system aspect in a systematic manner to evaluate thesystem aspect in various combinations of analysis metrics as part of anoverall system evaluation. The overall system evaluation may be done onetime, periodically, or continuously. As yet another example, theanalysis system determines the manner in which to evaluate the systemaspect as part of a systematic analysis of a section of the system,which may be done one time, periodically, or continuously.

Once the analysis system has determined the system aspect, how it is tobe evaluated, and the manner for evaluation, the data gathering function216 gathers data relevant to the system aspect, how it's to beevaluated, and the manner of evaluation from the system 11, fromresources that store system information 210 (e.g., from the system, froma private storage of the analysis system, etc.), and/or from one or moresystem proficiency resources 22. For example, a current evaluation isregarding an understanding (analysis perspective) of policies (analysismetric) to identify (analysis category) assets (evaluation mode) of anengineering department (system elements) regarding operations(evaluation criteria) that the assets perform based on discovered data(analysis viewpoint). As such, the data gathering function 216 gathersdata regarding policies to identify assets of the engineering departmentand the operations they perform using one or more data discovery tools.

The pre-processing function 217 processes the gathered data bynormalizing the data, parsing the data, tagging the data, and/orde-duplicating the data. The analysis system evaluations the processeddata in accordance with the selected analysis metric to produce one ormore ratings 219. For example, the analysis system would produce arating regarding the understanding of policies to identify assets of anengineering department regarding operations that the assets performbased on discovered data. The rating 219 is on a scale from low to high.In this example, a low rating indicates issues with the understandingand a high rating indicates no issues with the understanding.

FIG. 30 is a schematic block diagram of another example of thefunctioning of an analysis system 10 evaluating a system element undertest 91. The functioning of the analysis system includes a deficiencyperspective function 230, a deficiency evaluation viewpoint function 31,and an auto-correction function 233.

The deficiency perspective function 230 receives one or more ratings 219and may also receive the data used to generate the ratings 219. Fromthese inputs, the deficiency perspective function 230 determines whetherthere is an understanding issue, an implementation issue, and/or anoperation issue. For example, an understanding (analysis perspective)issue relates to a low understanding evaluation rating for a specificevaluation regarding policies (analysis metric) to identify (analysiscategory) assets (evaluation mode) of an engineering department (systemelements) regarding operations (evaluation criteria) that the assetsperform based on discovered data (analysis viewpoint).

As another example, an implementation (analysis perspective) issuerelates to a low implementation evaluation rating for a specificevaluation regarding implementation and/or use of policies (analysismetric) to identify (analysis category) assets (evaluation mode) of anengineering department (system elements) regarding operations(evaluation criteria) that the assets perform based on discovered data(analysis viewpoint). As yet another example, an operation (analysisperspective) issue relates to a low operation evaluation rating for aspecific evaluation regarding consistent, reliable, and/or accuratemechanism(s) to identify (analysis category) assets (evaluation mode) ofan engineering department (system elements) regarding operations(evaluation criteria) that the assets perform based on discovered data(analysis viewpoint) and on policies (analysis metric).

When an understanding, implementation, and/or operation issue isidentified, the deficiency evaluation viewpoint function 231 determineswhether the issue(s) is based on disclosed data, discovered data, and/ordesired data. For example, an understanding issue may be based on adifference between disclosed data and discovered data. As a specificexample, the disclosed data includes a policy outline how to identify(analysis category) assets (evaluation mode) of an engineeringdepartment (system elements) regarding operations (evaluation criteria)that the assets perform, which is listed as version 1.12 and a lastrevision date of Oct. 2, 2020. In this specific example, the discovereddata includes the same policy, but is has been updated to version 1.14and the last revision date as Nov. 13, 2020. As such, the deficiencyevaluation viewpoint function identifies a deficiency 232 in thedisclosed data as being an outdated policy.

As another specific example, the disclosed data includes a policyoutline how to identify (analysis category) assets (evaluation mode) ofan engineering department (system elements) regarding operations(evaluation criteria) that the assets perform. The disclosed data alsoshows an inconsistent use and/or application of the policy resulting oneor more assets not being properly identified. In this instance, thedeficiency evaluation viewpoint function identifies a deficiency 232 inthe disclosed data as being inconsistent use and/or application of thepolicy.

The auto-correct function 233 receives a deficiency 232 and interpretsit to determine a deficiency type, i.e., a nature of the understandingissue, the implementation issue, and/or the operation issues. Continuingwith the outdated policy example, the nature of the understanding issueis that there is a newer version of the policy. Since there is a newerversion available, the auto-correct function 233 can update the policyto the newer version for the system (e.g., an auto-correction). Inaddition to making the auto-correction 235, the analysis system createsan accounting 236 of the auto-correction (e.g., creates a record). Therecord includes an identity of the deficiency, date information, whatauto-correction was done, how it was done, verification that it wasdone, and/or more or less data as may be desired for recordingauto-corrections.

As another specific example, a deficiency 232 is discovered that anasset exists in the engineering department that was not included in thedisclosed data. This deficiency may include one or more relateddeficiencies. For example, a deficiency of design, a deficiency ofbuild, a deficiency is oversight of asset installation, etc. Thedeficiencies of design, build, and/or installation oversight can beauto-corrected; the deficiency of an extra asset cannot. With regard tothe deficiency of the extra asset, the analysis system generates areport regarding the extra asset and the related deficiencies.

FIG. 31 is a diagram of an example of evaluation options of an analysissystem 10 for evaluating a system element under test 91. The evaluationoptions are shown in a three-dimensional tabular form. The rows includeanalysis perspective 213 options (e.g., understanding, implementation,and operation). The columns includes analysis viewpoint 214 option(e.g., disclosed, discovered, and desired). The third dimension includesanalysis output 240 options (e.g., ratings 219, deficiencies indisclosed data, deficiencies in discovered data, deficiencies indisclosed to discovered data, deficiencies in disclosed to desired data,deficiencies in discovered to desired data, and auto-correct.

The analysis system 10 can evaluate the system element under test 91(e.g., system aspect) in one or more combinations of a row selection, acolumn selection, and/or a third dimension selection. For example, theanalysis system performs an evaluation from an understandingperspective, a disclosed data viewpoint, and a ratings output. Asanother example, the analysis system performs an evaluation from anunderstanding perspective, all viewpoints, and a ratings output.

FIG. 32 is a diagram of another example of evaluation options of ananalysis system 10 for evaluating a system element under test 91 (e.g.,system aspect). The evaluation options are shown in the form of a table.The rows are assets (physical and conceptual) and the columns are systemfunctions. The analysis system 10 can evaluate the system element undertest 91 (e.g., system aspect) in one or more combinations of a rowselection and a column selection.

For example, the analysis system 10 can evaluate user HW with respect tobusiness operations. As another example, the analysis system 10 canevaluate physical assets with respect to data flow. As another example,the analysis system 10 can evaluate user SW with respect to all systemfunctions.

FIG. 33 is a diagram of another example of evaluation options of ananalysis system 10 for evaluating a system element under test 91 (e.g.,system aspect). The evaluation options are shown in the form of a table.The rows are security functions and the columns are system functions.The analysis system 10 can evaluate the system element under test 91(e.g., system aspect) in one or more combinations of a row selection anda column selection.

For example, the analysis system 10 can evaluate threat detection withrespect to business operations. As another example, the analysis system10 can evaluate all security functions with respect to data flow. Asanother example, the analysis system 10 can evaluate threat avoidancewith respect to all system functions.

FIG. 34 is a diagram of another example of evaluation options of ananalysis system 10 for evaluating a system element under test 91 (e.g.,system aspect). The evaluation options are shown in the form of a table.The rows are assets (physical and conceptual) and the columns aresecurity functions. The analysis system 10 can evaluate the systemelement under test 91 (e.g., system aspect) in one or more combinationsof a row selection and a column selection.

For example, the analysis system 10 can evaluate user HW with respect tothreat recovery. As another example, the analysis system 10 can evaluatephysical assets with respect to threat resolution. As another example,the analysis system 10 can evaluate user SW with respect to all securityfunctions.

FIG. 35 is a schematic block diagram of an embodiment of an analysissystem 10 that includes one or more computing entities 16, one or moredatabases 275, one or more data extraction modules 80, one or moresystem user interface modules 81, and one or more remediation modules257. The computing entity(ies) 16 is configured to include a data inputmodule 250, a pre-processing module 251, a data analysis module 252, ananalytics modeling module 253, an evaluation processing module 254, adata output module 255, and a control module 256. The database 275,which includes one or more databases, stores the private data for aplurality of systems (e.g., systems A-x) and stores analytical data 270of the analysis system 10.

In an example, the system 11 provides input 271 to the analysis system10 via the system user interface module 80. The system user interfacemodule 80 provides a user interface for an administrator of the system11 and provides a s secure end-point of a secure data pipeline betweenthe system 11 and the analysis system 10. While the system userinterface module 81 is part of the analysis system, it is loaded on andis executed on the system 11.

Via the system user interface module 81, the administrator makesselections as to how the system is to be evaluated and the desiredoutput from the evaluation. For example, the administrator selectsevaluate system, which instructs the analysis system 10 to evaluate thesystem from most every, if not every, combination of system aspect(e.g., system element, system criteria, and system mode), evaluationaspect (e.g., evaluation perspective, evaluation viewpoint, andevaluation category), evaluation metric (e.g., process, policy,procedure, automation, documentation, and certification), and analysisoutput (e.g., an evaluation rating, deficiencies identified, andauto-correction of deficiencies). As another example, the administratorselects one or more system aspects, one or more evaluation aspects, oneor more evaluation metrics, and/or one or more analysis outputs.

The analysis system 10 receives the evaluation selections as part of theinput 271. A control module 256 interprets the input 271 to determinewhat part of the system is to be evaluated (e.g., system aspects), howthe system is to be evaluated (e.g., evaluation aspects), the manner inwhich the system is to be evaluated (e.g., evaluation metrics), and/orthe resulting evaluation output (e.g., an evaluation rating, adeficiency report, and/or auto-correction). From the interpretation ofthe input, the control module 256 generates data gathering parameters263, pre-processing parameters 264, data analysis parameters 265, andevaluation parameters 266.

The control module 256 provides the data gathering parameters 263 to thedata input module 250. The data input module 250 interprets the datagathering parameters 263 to determine data to gather. For example, thedata gathering parameters 263 are specific to the evaluation to beperformed by the analysis system 10. As a more specific example, if theanalysis system 10 is evaluating the understanding of the policies,processes, documentation, and automation regarding the assets built foran engineering department, then the data gathering parameters 263 wouldprescribe gathering data related to policies, processes, documentation,and automation regarding the assets built for the engineeringdepartment.

The data input module 250 may gather (e.g., retrieve, request, etc.)from a variety of sources. For example, the data input module 250gathers data 258 from the data extraction module 80. In this example,the data input module 250 provides instructions to the data extractionmodule 80 regarding the data being requested. The data extraction module80 pulls the requested data from system information 210, which may becentralized data of the system, system administration data, and/or datafrom assets of the system.

As another example, the data input module 250 gathers data from one ormore external data feeds 259. A source of an external data feed includesone or more business associate computing devices 23, one or morepublicly available servers 27, and/or one or more subscriber servers 28.Other sources of external data feeds 259 includes bot computing devices25, and/or bad actor computing devices 26. Typically, the data inputmodule 250 does not seek data inputs from bot computing devices 25and/or bad actor computing devices 26 except under certain circumstancesinvolving specific types of cybersecurity risks.

As another example, the data input module 250 gathers system proficiencydata 260 from one or more system proficiency resources 22. As a specificexample, for a data request that includes desired data, the data inputmodule 250 addresses one or more system proficiencies resources 22 toobtain the desired system proficiency data 260. For example, systemproficiency data 260 includes information regarding best-in-classpractices (for system requirements, for system design, for systemimplementation, and/or for system operation), governmental and/orregulatory requirements, security risk awareness and/or risk remediationinformation, security risk avoidance, performance optimizationinformation, system development guidelines, software developmentguideline, hardware requirements, networking requirements, networkingguidelines, and/or other system proficiency guidance.

As another example, the data input module 250 gathers stored data 261from the database 275. The stored data 261 is previously stored datathat is unique to the system 11, is data from other systems, ispreviously processed data, is previously stored system proficiency data,and/or is previously stored data that assists in the current evaluationof the system.

The data input module 250 provides the gathered data to thepre-processing module 251. Based on the pre-processing parameters 264(e.g., normalize, parse, tag, de-duplication, sort, filter, etc.), thepre-processing module 251 processes the gathered data to producepre-processed data 267. The pre-processed data 267 may be stored in thedatabase 275 and later retrieved as stored data 261.

The analysis modeling module 253 retrieves stored data 261 and/or storedanalytics 262 from the database 275. The analysis modeling module 253operates to increase the artificial intelligence of the analysis system10. For example, the analysis modeling module 253 evaluates stored datafrom one or more systems in a variety of ways to test the evaluationprocesses of the analysis system. As a more specific example, theanalysis modeling module 253 models the evaluation of understanding ofthe policies, processes, documentation, and automation regarding theassets built for an engineering department across multiple systems toidentify commonalities and/or deviations. The analysis modeling module253 interprets the commonalities and/or deviations to adjust parametersof the evaluation of understanding and models how the adjustments affectthe evaluation of understanding. If the adjustments have a positiveeffect, the analysis modeling module 253 stores them as analytics 262and/or analysis modeling 268 in the database 275.

The data analysis module 252 receives the pre-processed data 267, thedata analysis parameters 265 and may further receive optional analysismodeling data 268. The data analysis parameters 265 includes identify ofselected evaluation categories (e.g., identify, protect, detect,respond, and recover), identity of selected evaluation sub-categories,identify of selected evaluation sub-sub categories, identity of selectedanalysis metrics (e.g., process, policy, procedure, automation,certification, and documentation), grading parameters for the selectedanalysis metrics (e.g., a scoring scale for each type of analysismetric), identity of selected analysis perspective (e.g., understanding,implementation, operation, and self-analysis), and/or identity ofselected analysis viewpoint (e.g., disclosed, discovered, and desired).

The data analysis module 252 generates one or more ratings 219 for thepre-processed data 267 based on the data analysis parameters 265. Thedata analysis module 252 may adjust the generation of the one or morerating 219 based on the analysis modeling data 268. For example, thedata analysis module 252 evaluates the understanding of the policies,processes, documentation, and automation regarding the assets built foran engineering department based on the pre-processed data 267 to produceat least one evaluation rating 219.

Continuing with this example, the analysis modeling 268 is regarding theevaluation of understanding of the policies, processes, documentation,and automation regarding the assets built for an engineering departmentof a plurality of different organizations operating on a plurality ofdifferent systems. The modeling indicates that if processes are wellunderstood, the understanding of the policies is less significant in theoverall understanding. In this instance, the data analysis module 252may adjusts its evaluation rating of the understanding to a morefavorably rating if the pre-processed data 267 correlates with themodeling (e.g., good understanding of processes).

The data analysis module 252 provides the rating(s) 219 to the dataoutput module 255 and to the evaluation processing module 254. The dataoutput module 255 provides the rating(s) 219 as an output 269 to thesystem user interface module 81. The system user interface module 81provides a graphical rendering of the rating(s) 219.

The evaluation processing module 254 processes the rating(s) 219 basedon the evaluation parameters 266 to identify deficiencies 232 and/or todetermine auto-corrections 235. The evaluation parameters 266 provideguidance on how to evaluate the rating(s) 219 and whether to obtain data(e.g., pre-processed data, stored data, etc.) to assist in theevaluation. The evaluation guidance includes how deficiencies are to beidentified. For example, identify the deficiencies based on thedisclosed data, based on the discovered data, based on a differencesbetween the disclosed and discovered data, based on a differencesbetween the disclosed and desired data, and/or based on a differencesbetween the discovered and desired data. The evaluation guidance furtherincludes whether auto-correction is enabled. The evaluation parameters266 may further includes deficiency parameters, which provide a level oftolerance between the disclosed, discovered, and/or desired data whendetermining deficiencies.

The evaluation processing module 254 provides deficiencies 232 and/orthe auto-corrections 235 to the data output module 255. The data outputmodule 255 provides the deficiencies 232 and/or the auto-corrections 235as an output 269 to the system user interface module 81 and to theremediation module 257. The system user interface module 81 provides agraphical rendering of the deficiencies 232 and/or the auto-corrections235.

The remediation module 257 interprets the deficiencies 232 and theauto-corrections 235 to identify auto-corrections to be performed withinthe system. For example, if a deficiency is a computing device having anoutdated user software application, the remediation module 257coordinates obtaining a current copy of the user software application,uploading it on the computing device, and updating maintenance logs.

FIG. 36 is a schematic block diagram of an embodiment of a portion of ananalysis system 10 coupled to a portion of the system 11. In particular,the data output module 255 of the analysis system 10 is coupled to aplurality of remediation modules 257-1 through 257-n. Each remediationmodule 257 is coupled to one or more system assets 280-1 through 280-n.

A remediation module 257 receives a corresponding portion of the output269. For example, remediation module 257-1 receives output 269-1, whichis regarding an evaluation rating, deficiency, and/or an auto-correctionof system asset 280-1. Remediation module 257-1 may auto-correct adeficiency of the system asset or a system element thereof.Alternatively or in addition, the remediation module 257-1 mayquarantine the system asset or system element thereof if the deficiencycannot be auto-corrected and the deficiency exposes the system toundesired risks, undesired liability, and/or undesired performancedegradation.

FIG. 37 is a schematic block diagram of another embodiment of a portionof an analysis system 10 coupled to a portion of the system 11. Inparticular, the data input module 250 of the analysis system 10 iscoupled to a plurality of data extraction modules 80-1 through 80-n.Each data extraction module 80 is coupled to a system data source 290 ofthe system 11. Each of the system data sources produce systeminformation 210 regarding a corresponding portion of the system. Asystem data source 290-1 through 290-n may be an Azure EventHub, CiscoAdvanced Malware Protection (AMP), Cisco Email Security Appliance (ESA),Cisco Umbrella, NetFlow, and/or Syslog. In addition, a system datasource may be a system asset, a system element, and/or a storage devicestoring system information 210.

An extraction data migration module 293 coordinates the collection ofsystem information 210 as extracted data 291-1 through 291-n. Anextraction data coordination module 292 coordinates the forwarding ofthe extracted data 291 as data 258 to the data input module 250.

FIG. 38 is a schematic block diagram of an embodiment of a dataextraction module 80 of an analysis system 10 coupled to a system 11.The data extraction module 80 includes a tool one or more interfacemodules 311, one or more processing module 312, and one or more networkinterfaces 313. The network interface 313 provides a network connectionsthat allows the data extraction module 80 to be coupled to the one ormore computing entities 16 of the analysis system 10. The tool interface311 allows the data extraction module 80 to interact with tools of thesystem 11 to obtain system information from system data sources 290.

The system 11 includes one or more tools that can be accessed by thedata extraction module 80 to obtain system information from one or moredata sources 290-1 through 290-n. The tools include one or more datasegmentation tools 300, one or more boundary detection tools 301, one ormore data protection tools 302, one or more infrastructure managementtools 303, one or more encryption tools 304, one or more exploitprotection tools 305, one or more malware protection tools 306, one ormore identity management tools 307, one or more access management tools308, one or more system monitoring tools, and/or one or morevulnerability management tools 310.

A system tool may also be an infrastructure management tool, a networkmonitoring tool, a network strategy and planning tool, a networkmanaging tool, a Simple Network Management Protocol (SNMP) tool, atelephony monitoring tool, a firewall monitoring tool, a bandwidthmonitoring tool, an IT asset inventory management tool, a networkdiscovery tool, a network asset discovery tool, a software discoverytool, a security discovery tool, an infrastructure discovery tool,Security Information & Event Management (SIEM) tool, a data crawlertool, and/or other type of tool to assist in discovery of assets,functions, security issues, implementation of the system, and/oroperation of the system.

Depending on the data gathering parameters, the tool interface 311engages a system tool to retrieve system information. For example, thetool interface 311 engages the identity management tool to identifyassets in the engineering department. The processing module 312coordinates requests from the analysis system 10 and responds to theanalysis system 10.

FIG. 39 is a schematic block diagram of another embodiment of ananalysis system 10 that includes one or more computing entities 16, oneor more databases 275, one or more data extraction modules 80, and oneor more system user interface modules 81. The computing entity(ies) 16is configured to include a data input module 250, a pre-processingmodule 251, a data analysis module 252, an analytics modeling module253, a data output module 255, and a control module 256. The database275, which includes one or more databases, stores the private data for aplurality of systems (e.g., systems A-x) and stores analytical data 270of the analysis system 10.

This embodiment operates similarly to the embodiment of FIG. 35 with theremoval of the evaluation module 254, which produces deficiencies 232and auto-corrections 235, and the removal of the remediation modules257. As such, this analysis system 10 produces evaluation ratings 219 asthe output 269.

FIG. 40 is a schematic block diagram of another embodiment of ananalysis system 10 that is similar to the embodiment of FIG. 39. Thisembodiment does not include a pre-processing module 251. As such, thedata collected by the data input module 250 is provided directly to thedata analysis module 252.

FIG. 41 is a schematic block diagram of an embodiment of a data analysismodule 252 of an analysis system 10. The data analysis module 252includes a data module 321 and an analysis & score module 336. The datamodule 321 includes a data parse module 320, one or more data storagemodules 322-334, and a source data matrix 335. A data storage module322-334 may be implemented in a variety of ways. For example, a datastorage module is a buffer. As another example, a data storage module isa section of memory (45, 56, 57, and/or 62 of the FIG. 2 series) of acomputing device (e.g., an allocated, or ad hoc, addressable section ofmemory). As another example, a data storage module is a storage unit(e.g., a computing device used primarily for storage). As yet anotherexample, a data storage module is a section of a database (e.g., anallocated, or ad hoc, addressable section of a database).

The data module 321 operates to provide the analyze & score module 336with source data 337 selected from incoming data based on one or moredata analysis parameters 265. The data analysis parameter(s) 265indicate(s) how the incoming data is to be parsed (if at all) and how itis to be stored within the data storage modules 322-334. A data analysisparameter 265 includes system aspect storage parameters 345, evaluationaspect storage parameters 346, and evaluation metric storage parameters347. A system aspect storage parameter 345 may be null or includesinformation to identify one or more system aspects (e.g., systemelement, system criteria, and system mode), how the data relating tosystem aspects is to be parsed, and how the system aspect parsed data isto be stored.

An evaluation aspect storage parameter 346 may be null or includesinformation to identify one or more evaluation aspects (e.g., evaluationperspective, evaluation viewpoint, and evaluation category), how thedata relating to evaluation aspects is to be parsed, and how theevaluation aspect parsed data is to be stored. An evaluation metricstorage parameter 347 may be null or includes information to identifyone or more evaluation metrics (e.g., process, policy, procedure,certification, documentation, and automation), how the data relating toevaluation metrics is to be parsed, and how the evaluation metric parseddata is to be stored. Note that the data module 321 interprets the dataanalysis parameters 265 collectively such that parsing and storage areconsistent with the parameters.

The data parsing module 320 parses incoming data in accordance with thesystem aspect storage parameters 345, evaluation aspect storageparameters 346, and evaluation metric storage parameters 347, whichgenerally correspond to what part of the system is being evaluation, howthe system is being evaluated, the manner of evaluation, and/or adesired analysis output. As such, incoming data may be parsed in avariety of ways. The data storage modules 322-334 are assigned to storeparsed data in accordance with the storage parameters 345-347. Forexample, the incoming data, which includes pre-processed data 267, otherexternal feed data 259, data 258 received via a data extraction module,stored data 261, and/or system proficiency data 260, is parsed based onsystem criteria (of the system aspect) and evaluation viewpoint (of theevaluation aspect). As a more specific example, the incoming data isparsed into, and stored, as follows:

-   -   disclosed guideline data that is stored in a disclosed guideline        data storage module 322;    -   discovered guideline data that is stored in a discovered        guideline data storage module 323;    -   desired guideline data that is stored in a desired guideline        data storage module 324;    -   disclosed system requirement (sys. req.) data that is stored in        a disclosed system requirement data storage module 325;    -   discovered system requirement (sys. req.) data that is stored in        a discovered system requirement data storage module 326;    -   desired system requirement (sys. req.) data that is stored in a        desired system requirement data storage module 327;    -   disclosed design and/or build data that is stored in a disclosed        design and/or build data storage module 328;    -   discovered design and/or build data that is stored in a        discovered design and/or build data storage module 329;    -   desired design and/or build data that is stored in a desired        design and/or build data storage module 330;    -   disclosed system operation data that is stored in a disclosed        system operation data storage module 331;    -   discovered system operation data that is stored in a discovered        system operation data storage module 332;    -   desired system operation data that is stored in a desired system        operation data storage module 333; and/or    -   other data that is stored in another data storage module 334.

As another example of parsing, the incoming data is parsed based on acombination of one or more system aspects (e.g., system elements, systemcriteria, and system mode) or sub-system aspects thereof, one or moreevaluation aspects (e.g., evaluation perspective, evaluation viewpoint,and evaluation category) or sub-evaluation aspects thereof, and/or oneor more evaluation rating metrics (e.g., process, policy, procedure,certification, documentation, and automation) or sub-evaluation ratingmetrics thereof. As a specific example, the incoming data is parsedbased on the evaluation rating metrics, creating processed parsed data,policy parsed data, procedure parsed data, certification parsed data,documentation parsed data, and automation parsed data. As anotherspecific example, the incoming data is parsed based on the evaluationcategory of identify and its sub-categories of asset management,business environment, governance, risk assessment, risk management,access control, awareness &, training, and/or data security.

As another example of parsing, the incoming data is not parsed, or isminimally parsed. As a specific example, the data is parsed based ontimestamps: data from one time period (e.g., a day) is parsed from dataof another time period (e.g., a different day).

The source data matrix 335, which may be a configured processing module,retrieves source data 337 from the data storage modules 322-334. Theselection corresponds to the analysis being performed by the analyze &score module 336. For example, if the analyze & score module 336 isevaluating the understanding of the policies, processes, documentation,and automation regarding the assets built for the engineeringdepartment, then the source data 337 would be data specific to policies,processes, documentation, and automation regarding the assets built forthe engineering department.

The analyze & score module 336 generates one or more ratings 219 for thesource data 337 in accordance with the data analysis parameters 265 andanalysis modeling 268. The data analysis parameters 265 includes systemaspect analysis parameters 342, evaluation aspect analysis parameters343, and evaluation metric analysis parameters 344. The analyze & scoremodule 336 is discussed in greater detail with reference to FIG. 42.

FIG. 42 is a schematic block diagram of an embodiment of an analyze andscore module 336 includes a matrix module 341 and a scoring module 348.The matrix module 341 processes an evaluation mode matrix, an evaluationperspective matrix, an evaluation viewpoint matrix, and an evaluationcategories matrix to produce a scoring input. The scoring module 348includes an evaluation metric matrix to process the scoring input datain accordance with the analysis modeling 268 to produce the rating(s)219.

For example, the matrix module 341 configures the matrixes based on thesystem aspect analysis parameters 342 and the evaluation aspect analysisparameters 343 to process the source data 337 to produce the scoringinput data. As a specific example, the system aspect analysis parameters342 and the evaluation aspect analysis parameters 343 indicate assets asthe evaluation mode, understanding as the evaluation perspective,discovered as the evaluation viewpoint, and the identify as theevaluation category.

Accordingly, the matrix module 341 communicates with the source datamatrix module 335 of the data module 321 to obtain source data 337relevant to assets, understanding, discovered, and identify. The matrixmodule 341 may organize the source data 337 using an organization scheme(e.g., by asset type, by evaluation metric type, by evaluationsub-categories, etc.) or keep the source data 337 as a collection ofdata. The matrix module 341 provides the scoring input data 344 as acollection of data or as organized data to the scoring module 348.

Continuing with the example, the scoring module 248 receives the scoringinput data 348 and evaluates in accordance with the evaluation metricanalysis parameters 344 and the analysis modeling 268 to produce therating(s) 219. As a specific example, the evaluation metric analysisparameters 344 indicate analyzing the scoring input data with respect toprocesses. In this instance, the analysis modeling 268 provides ascoring mechanism for evaluating the scoring input data with respect toprocesses to the scoring module 248. For instance, the analysis modeling268 includes six levels regarding processes and a correspondingnumerical rating: none (e.g., 0), inconsistent (e.g., 10), repeatable(e.g., 20), standardized (e.g., 30), measured (e.g., 40), and optimized(e.g., 50).

In addition, the analysis modeling 268 includes analysis protocols forinterpreting the scoring input data to determine its level andcorresponding rating. For example, if there are no processes regardingidentifying assess of the discovered data, then an understanding levelof processes would be none (e.g., 0), since there are no processes. Asanother example, if there are some processes regarding identifyingassess of the discovered data, but there are gaps in the processes(e.g., identifies some assets, but not all, do not produce consistentresults), then an understanding level of processes would be inconsistent(e.g., 10). To determine if there are gaps in the processes, the scoremodule 248 executes the processes of the discovered data to identifyassets. The scoring module 248 also executes one or more asset discoverytools to identify assets and then compares the two results. If there areinconsistencies in the identified assets, then there are gaps in theprocesses.

As a further example, the processes regarding identifying assess of thediscovered data are repeatable (e.g., produces consistent results, butthere are variations in the processes from process to process, and/orthe processes are not all regulated) but not standardized (e.g.,produces consistent results, but there are no appreciable variations inthe processes from process to process, and/or the processes areregulated). If the processes are repeatable but not standardized, thescoring module establishes an understanding level of the processes asrepeatable (e.g., 20).

If the processes are standardized, the scoring module then determineswhether the processes are measured (e.g., precise, exact, and/orcalculated to the task of identifying assets). If not, the scoringmodule establishes an understanding level of the processes asstandardized (e.g., 30).

If the processes are measured, the scoring module then determineswhether the processes are optimized (e.g., up-to-date and improvementassessed on a regular basis as part of system protocols). If not, thescoring module establishes an understanding level of the processes asmeasured (e.g., 40). If so, the scoring module establishes anunderstanding level of the processes as optimized (e.g., 50).

FIG. 43 is a diagram of an example of system aspect, evaluation aspect,evaluation rating metric, and analysis system output options of ananalysis system 10 for analyzing a system 11, or portion thereof. Thesystem aspect corresponds to what part of the system is to be evaluatedby the analysis system. The evaluation aspect indicates how the systemaspect is to be evaluation. The evaluation rating metric indicates themanner of evaluation of the system aspect in accordance with theevaluation aspect. The analysis system output indicates the type ofoutput to be produced by the analysis system based on the evaluation ofthe system aspect in accordance with the evaluation aspect as per theevaluation rating metric.

The system aspect includes system elements, system criteria, and systemmodes. A system element includes one or more system assets, which is aphysical asset and/or a conceptual asset. For example, a physical assetis a computing entity, a computing device, a user software application,a system software application (e.g., operating system, etc.), a softwaretool, a network software application, a security software application, asystem monitoring software application, and the like. As anotherexample, a conception asset is a hardware architecture (e.g.,identification of a system's physical components, their capabilities,and their relationship to each other) and/or sub-architectures thereofand a software architecture (e.g., fundamental structures for thesystem's software, their requirements, and inter-relational operations)and sub-architectures thereof.

A system element and/or system asset is identifiable in a variety ofways. For example, it can be identified by an organization identifier(ID), which would be associated with most, if not all, system elementsof a system. As another example, a system element and/or system assetcan be identified by a division ID, where the division is one of aplurality of divisions in the organization. As another example, a systemelement and/or system asset can be identified by a department ID, wherethe department is one of a plurality of departments in a division. Asyet another example, a system element and/or system asset can beidentified by a department ID, where the department is one of aplurality of departments in a division. As a further example, a systemelement and/or system asset can be identified by a group ID, where thedepartment is one of a plurality of groups in a department. As a stillfurther example, a system element and/or system assets can be identifiedby a sub-group ID, where the department is one of a plurality ofsub-groups in a group. With this type of identifier, a collection ofsystem elements and/or system assets can be selected for evaluation byusing an organization ID, a division ID, a department ID, a group ID, ora sub-group ID.

A system element and/or system asset may also be identified based on auser ID, a serial number, vendor data, an IP address, etc. For example,a computing device has a serial number and vendor data. As such, thecomputing device can be identified for evaluation by its serial numberand/or the vendor data. As another example, a software application has aserial number and vendor data. As such, the software application can beidentified for evaluation by its serial number and/or the vendor data.

In addition, an identifier of one system element and/or system asset maylink to one or more other system elements and/or system assets. Forexample, computing device has a device ID, a user ID, and/or a serialnumber to identify it. The computing device also includes a plurality ofsoftware applications, each with its own serial number. In this example,the software identifiers are linked to the computing device identifiersince the software is loaded on the computing device. This type of anidentifier allows a single system asset to be identified for evaluation.

The system criteria includes information regarding the development,operation, and/or maintenance of the system 11. For example, a systemcriteria is a guideline, a system requirement, a system designcomponent, a system build component, the system, and system operation.Guidelines, system requirements, system design, system build, and systemoperation were discussed with reference to FIG. 25.

The system mode indicates the assets of the system, the system functionsof the system, and/or the security functions of the system are to beevaluated. Assets, system functions, and security functions have beenpreviously discussed with reference to one or more of FIGS. 7-24 and32-34.

The evaluation aspect, which indicates how the system aspect is to beevaluated, includes evaluation perspective, evaluation viewpoint, andevaluation category. The evaluation perspective includes understanding(e.g., how well the system is known, should be known, etc.);implementation, which includes design and/or build, (e.g., how well isthe system designed, how well should it be designed); systemperformance, and/or system operation (e.g., how well does the systemperform and/or operate, how well should it perform and/or operate); andself-analysis (e.g., how self-aware is the system, how self-healing isthe system, how self-updating is the system).

The evaluation viewpoint includes disclosed data, discovered data, anddesired data. Disclosed data is the known data of the system at theoutset of an analysis, which is typically supplied by a systemadministrator and/or is obtained from data files of the system.Discovered data is the data discovered about the system by the analysissystem during the analysis. Desired data is the data obtained by theanalysis system from system proficiency resources regarding desiredguidelines, system requirements, system design, system build, and/orsystem operation. Differences in disclosed, discovered, and desired dataare evaluated to support generating an evaluation rating, to identifydeficiencies, and/or to determine and provide auto-corrections.

The evaluation category includes an identify category, a protectcategory, a detect category, a respond category, and a recover category.In general, the identify category is regarding identifying assets,system functions, and/or security functions of the system; the protectcategory is regarding protecting assets, system functions, and/orsecurity functions of the system from issues that may adversely affect;the detect category is regarding detecting issues that may, or have,adversely affect assets, system functions, and/or security functions ofthe system; the respond category is regarding responding to issues thatmay, or have, adversely affect assets, system functions, and/or securityfunctions of the system; and the recover category is regardingrecovering from issues that have adversely affect assets, systemfunctions, and/or security functions of the system. Each categoryincludes one or more sub-categories and each sub-category may includeone or more sub-sub categories as discussed with reference to FIGS.44-49.

The evaluation rating metric includes process, policy, procedure,certification, documentation, and automation. The evaluation ratingmetric may include more or less topics. The analysis system outputoptions include evaluation rating, deficiency identification, anddeficiency auto-correction.

With such a significant number of options with the system aspect, theevaluation aspect, the evaluation rating metrics, and analysis systemoutput options, the analysis system can analyze a system in thousands,or more, combinations. For example, the analysis system 10 could providean evaluation rating for the entire system with respect to itsvulnerability to cyber-attacks. The analysis system 10 could alsoidentify deficiencies in the system's cybersecurity processes, policies,documentation, implementation, operation, assets, and/or securityfunctions based on the evaluation rating. The analysis system 10 couldfurther auto-correct at least some of the deficiencies in the system'scybersecurity processes, policies, documentation, implementation,operation, assets, and/or security functions.

As another example, the analysis system 10 could evaluates the system'srequirements for proper use of software (e.g., authorized to use, validcopy, current version) by analyzing every computing device in the systemas to the system's software use requirements. From this analysis, theanalysis system generates an evaluation rating. The analysis system 10could also identify deficiencies in the compliance with the system'ssoftware use requirements (e.g., unauthorized use, invalid copy,outdated copy). The analysis system 10 could further auto-correct atleast some of the deficiencies in compliance with the system's softwareuse requirements (e.g., remove invalid copies, update outdated copies).

FIG. 44 is a diagram of another example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11. This diagram is similar to FIG. 43 with theexception that this figure illustrates sub-categories and sub-subcategories. Each evaluation category includes sub-categories, which, inturn, include their own sub-sub categories. The various categories,sub-categories, and sub-sub categories corresponds to the categories,sub-categories, and sub-sub categories identified in the “Framework forImproving Critical Instructure Cybersecurity”, Version 1.1, Apr. 16,2018 by the National Institute of Standards and Technology (NIST).

FIG. 45 is a diagram of an example of an identification evaluationcategory that includes a plurality of sub-categories and eachsub-category includes its own plurality of sub-sub-categories. Theidentify category includes the sub-categories of asset management,business environment, governance, risk management, access control,awareness & training, and data security.

The asset management sub-category includes the sub-sub categories of HWinventoried, SW inventoried, data flow mapped out, external systemscataloged, resources have been prioritized, and security roles have beenestablished. The business environment sub-category includes the sub-subcategories of supply chain roles defined, industry criticalinfrastructure identified, business priorities established, criticalservices identified, and resiliency requirements identified.

The governance sub-category includes the sub-sub categories of securitypolicies are established, security factors aligned, and legalrequirements are identified. The risk assessment sub-category includesthe sub-sub categories of vulnerabilities identified, external sourcesare leveraged, threats are identified, business impacts are identified,risk levels are identified, and risk responses are identified. The riskmanagement sub-category includes the sub-sub categories of riskmanagement processes are established, risk tolerances are established,and risk tolerances are tied to business environment.

The access control sub-category includes the sub-sub categories ofremote access control is defined, permissions are defined, and networkintegrity is defined. The awareness & training sub-category includes thesub-sub categories of users are trained, user privileges are known,third party responsibilities are known, executive responsibilities areknown, and IT and security responsibilities are known. The data securitysub-category includes the sub-sub categories of data at rest protocolsare established, data in transit protocols are established, formal assetmanagement protocols are established, adequate capacity of the system isestablished, data leak prevention protocols are established, integritychecking protocols are established, and use and development separationprotocols are established.

FIG. 46 is a diagram of an example of a protect evaluation category thatincludes a plurality of sub-categories and each sub-category includesits own plurality of sub-sub-categories. The protect category includesthe sub-categories of information protection processes and procedures,maintenance, and protective technology.

The information protection processes and procedures sub-categoryincludes the sub-sub categories of baseline configuration ofIT/industrial controls are established, system life cycle management isestablished, configuration control processes are established, backups ofinformation are implemented, policy & regulations for physical operationenvironment are established, improving protection processes areestablished, communication regarding effective protection technologiesis embraced, response and recovery plans are established, cybersecurityin is including in human resources, and vulnerability management plansare established.

The maintenance sub-category includes the sub-sub categories of systemmaintenance & repair of organizational assets programs are establishedand remote maintenance of organizational assets is established. Theprotective technology sub-category includes the sub-sub-categories ofaudit and recording policies are practiced, removable media is protected& use policies are established, access to systems and assets iscontrolled, and communications and control networks are protected.

FIG. 47 is a diagram of an example of a detect evaluation category thatincludes a plurality of sub-categories and each sub-category includesits own plurality of sub-sub-categories. The detect category includesthe sub-categories of anomalies and events, security continuousmonitoring, and detection processes.

The anomalies and events sub-category includes the sub-sub categories ofbaseline of network operations and expected data flows are monitored,detected events are analyzed, event data are aggregated and correlated,impact of events is determined, and incident alert thresholds areestablished. The security continuous monitoring sub-category includesthe sub-sub categories of network is monitored to detect potentialcybersecurity attacks, physical environment is monitored forcybersecurity events, personnel activity is monitored for cybersecurityevents, malicious code is detected, unauthorized mobile codes isdetected, external service provider activity is monitored forcybersecurity events, monitoring for unauthorized personnel,connections, devices, and software is performed, and vulnerability scansare performed. The detection processes sub-category includes the sub-subcategories of roles and responsibilities for detection are defined,detection activities comply with applicable requirements, detectionprocesses are tested, event detection information is communicated, anddetection processes are routinely improved.

FIG. 48 is a diagram of an example of a respond evaluation category thatincludes a plurality of sub-categories and each sub-category includesits own plurality of sub-sub-categories. The respond category includesthe sub-categories of response planning, communications, analysis,mitigation, and improvements.

The response planning sub-category includes the sub-sub category ofresponse plan is executed during and/or after an event. Thecommunications sub-category includes the sub-sub category of personnelroles and order of operation are established, events are reportedconsistent with established criteria, information is shared consistentlyper the response plan, coordination with stakeholders is consistent withthe response plan, and voluntary information is shared with externalstakeholders.

The analysis sub-category includes the sub-sub categories ofnotifications form detection systems are investigated, impact of theincident is understood, forensics are performed, and incidents arecategorized per response plan. The mitigation sub-category includes thesub-sub categories of incidents are contained, incidents are mitigated,and newly identified vulnerabilities are processed. The improvementssub-categories includes the sub-sub categories of response plansincorporate lessons learned, and response strategies are updated.

FIG. 49 is a diagram of an example of a recover evaluation category thatincludes a plurality of sub-categories and each sub-category includesits own plurality of sub-sub-categories. The recover category includesthe sub-categories of recovery plan, improvements, and communication.The recovery plan sub-category includes the sub-sub category of recoveryplan is executed during and/or after an event.

The improvement sub-category includes the sub-sub categories of recoveryplans incorporate lessons learned and recovery strategies are updated.The communications sub-category includes the sub-sub categories ofpublic relations are managed, reputations after an event is repaired,and recovery activities are communicated.

FIG. 50 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the understanding of theguidelines for identifying assets, protecting the assets from issues,detecting issues that may affect or are affecting the assets, respondingto issues that may affect or are affecting the assets, and recoveringfrom issues that affected the assets of a department based on discloseddata.

For this specific example, the analysis system 10 obtains disclosed datafrom the system regarding the guidelines associated with the assets ofthe department. From the disclosed data, the analysis system renders anevaluation rating for the understanding of the guidelines foridentifying assets. The analysis system renders a second evaluationrating for the understanding of the guidelines regarding protection ofthe assets from issues. The analysis system renders a third evaluationrating for the understanding of the guidelines regarding detection ofissues that may affect or are affecting the assets.

The analysis system renders a fourth evaluation rating for theunderstanding of the guidelines regarding responds to issues that mayaffect or are affecting the assets. The analysis system renders a fifthevaluation rating for the understanding of the guidelines regardingrecovery from issues that affected the assets of a department based ondisclosed data. The analysis system may render an overall evaluationrating for the understanding of the guidelines based on the firstthrough fifth evaluation ratings.

As another example, the analysis system 11 evaluates the understandingof guidelines used to determine what assets should be included in thedepartment, how the assets should be protected from issues, how issuesthat may affect or are affecting the assets are detect, how to respondto issues that may affect or are affecting the assets, and how theassets will recover from issues that may affect or are affecting thembased on disclosed data. In this example, the analysis system renders anevaluation rating for the understanding of the guidelines regarding whatassets should be in the department. The analysis system renders a secondevaluation rating for the understanding of the guidelines regarding howthe assets should be protected from issues. The analysis system rendersa third evaluation rating for the understanding of the guidelinesregarding how to detect issues that may affect or are affecting theassets.

The analysis system renders a fourth evaluation rating for theunderstanding of the guidelines regarding how to respond to issues thatmay affect or are affecting the assets. The analysis system renders afifth evaluation rating for the understanding of the guidelinesregarding how to recover from issues that affected the assets of adepartment based on disclosed data. The analysis system may render anoverall evaluation rating for the understanding based on the firstthrough fifth evaluation ratings.

FIG. 51 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the understanding of thesystem design for identifying assets, protecting the assets from issues,detecting issues that may affect or are affecting the assets, respondingto issues that may affect or are affecting the assets, and recoveringfrom issues that affected the assets of a department based on discloseddata.

For this specific example, the analysis system 10 obtains disclosed datafrom the system regarding the system design associated with the assetsof the department. From the disclosed data, the analysis system rendersan evaluation rating for the understanding of the system design foridentifying assets. The analysis system renders a second evaluationrating for the understanding of the system design regarding protectionof the assets from issues. The analysis system renders a thirdevaluation rating for the understanding of the system design regardingdetection of issues that may affect or are affecting the assets.

The analysis system renders a fourth evaluation rating for theunderstanding of the system design regarding responds to issues that mayaffect or are affecting the assets. The analysis system renders a fifthevaluation rating for the understanding of the system design regardingrecovery from issues that affected the assets of a department based ondisclosed data. The analysis system may render an overall evaluationrating for the understanding based on the first through fifth evaluationratings.

As another example, the analysis system 11 evaluates the understandingof system design used to determine what assets should be included in thedepartment, how the assets should be protected from issues, how issuesthat may affect or are affecting the assets are detect, how to respondto issues that may affect or are affecting the assets, and how theassets will recover from issues that may affect or are affecting thembased on disclosed data. In this example, the analysis system renders anevaluation rating for the understanding of the system design regardingwhat assets should be in the department. The analysis system renders asecond evaluation rating for the understanding of the system designregarding how the assets should be protected from issues. The analysissystem renders a third evaluation rating for the understanding of thesystem design regarding how to detect issues that may affect or areaffecting the assets.

The analysis system renders a fourth evaluation rating for theunderstanding of the system design regarding how to respond to issuesthat may affect or are affecting the assets. The analysis system rendersa fifth evaluation rating for the understanding of the system designregarding how to recover from issues that affected the assets of adepartment based on disclosed data. The analysis system may render anoverall evaluation rating for the understanding based on the firstthrough fifth evaluation ratings.

FIG. 52 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the understanding of theguidelines, system requirements, and system design for identifyingassets, protecting the assets from issues, detecting issues that mayaffect or are affecting the assets, responding to issues that may affector are affecting the assets, and recovering from issues that affectedthe assets of a department based on disclosed data and discovered data.

For this specific example, the analysis system 10 obtains disclosed dataand discovered from the system regarding guidelines, systemrequirements, and system design associated with the assets of thedepartment. From the disclosed data and discovered data, the analysissystem renders one or more first evaluation ratings (e.g., one for eachof guidelines, system requirements, and system design, or one for allthree) for the understanding of the guidelines, system requirements, andsystem design for identifying assets. The analysis system renders one ormore second evaluation ratings for the understanding of the guidelines,system requirements, and system design regarding protection of theassets from issues. The analysis system renders one or more thirdevaluation ratings for the understanding of the guidelines, systemrequirements, and system design regarding detection of issues that mayaffect or are affecting the assets.

The analysis system renders one or more fourth evaluation ratings forthe understanding of the guidelines, system requirements, and systemdesign regarding responds to issues that may affect or are affecting theassets. The analysis system renders one or more fifth evaluation ratingsfor the understanding of the guidelines, system requirements, and systemdesign regarding recovery from issues that affected the assets of adepartment based on disclosed data. The analysis system may render anoverall evaluation rating for the understanding based on the one or morefirst through one or more fifth evaluation ratings.

The analysis system 11 may further render an understanding evaluationrating regarding how well the discovered data correlates with thedisclosed data. In other words, evaluate the knowledge level of thesystem. In this example, the analysis system compares the disclosed datawith the discovered data. If they substantially match, the understandingof the system would receive a relatively high evaluation rating. Themore the disclosed data differs from the discovered data, the lower theunderstanding evaluation rating will be.

As another example, the analysis system 11 evaluates the understandingof guidelines, system requirements, and system design used to determinewhat assets should be included in the department, how the assets shouldbe protected from issues, how issues that may affect or are affectingthe assets are detect, how to respond to issues that may affect or areaffecting the assets, and how the assets will recover from issues thatmay affect or are affecting them based on disclosed data and discovereddata. In this example, the analysis system renders one or more firstevaluation ratings for the understanding of the guidelines, systemrequirements, and system design regarding what assets should be in thedepartment. The analysis system renders one or more second evaluationratings for the understanding of the guidelines, system requirements,and system design regarding how the assets should be protected fromissues. The analysis system renders one or more third evaluation ratingsfor the understanding of the guidelines, system requirements, and systemdesign regarding how to detect issues that may affect or are affectingthe assets.

The analysis system renders one or more fourth evaluation ratings forthe understanding of the guidelines, system requirements, and systemdesign regarding how to respond to issues that may affect or areaffecting the assets. The analysis system renders one or more fifthevaluation ratings for the understanding of the guidelines, systemrequirements, and system design regarding how to recover from issuesthat affected the assets of a department based on disclosed data. Theanalysis system may render an overall evaluation rating for theunderstanding of the guidelines, system requirements, and system designbased on the one or more first through the one or more fifth evaluationratings.

FIG. 53 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the implementation forand operation of identifying assets of a department, protecting theassets from issues, detecting issues that may affect or are affectingthe assets, responding to issues that may affect or are affecting theassets, and recovering from issues that affected the assets per theguidelines, system requirements, system design, system build, andresulting system based on disclosed data and discovered data.

For this specific example, the analysis system 10 obtains disclosed dataand discovered data from the system regarding the guidelines, systemrequirements, system design, system build, and resulting systemassociated with the assets of the department. From the disclosed dataand discovered data, the analysis system renders one or more firstevaluation ratings (e.g., one for each of guidelines, systemrequirements, system design, system build, resulting system with respectto each of implementation and operation or one for all of them) for theimplementation and operation of identifying the assets per theguidelines, system requirements, system design, system build, andresulting system. The analysis system renders one or more secondevaluation ratings for the implementation and operation of protectingthe assets from issues per the guidelines, system requirements, systemdesign, system build, and resulting system.

The analysis system renders one or more third evaluation ratings for theimplementation and operation of detecting issues that may affect or areaffecting the assets per the guidelines, system requirements, systemdesign, system build, and resulting system. The analysis system rendersone or more fourth evaluation ratings for the implementation andoperation of responding to issues that may affect or are affecting theassets per the guidelines, system requirements, system design, systembuild, and resulting system.

The analysis system renders one or more fifth evaluation ratings for theimplementation and operation of recovering from issues that may affector are affecting the assets per the guidelines, system requirements,system design, system build, and resulting system. The analysis systemmay render an overall evaluation rating for the implementation and/orperformance based on the one or more first through one or more fifthevaluation ratings.

FIG. 54 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the implementation forand operation of identifying assets of a department, protecting theassets from issues, detecting issues that may affect or are affectingthe assets, responding to issues that may affect or are affecting theassets, and recovering from issues that affected the assets per theguidelines, system requirements, system design, system build, andresulting system based on discovered data and desired data.

For this specific example, the analysis system 10 obtains disclosed dataand discovered from the system regarding the guidelines, systemrequirements, system design, system build, and resulting systemassociated with the assets of the department. From the discovered dataand desired data, the analysis system renders one or more firstevaluation ratings (e.g., one for each of guidelines, systemrequirements, system design, system build, resulting system with respectto each of implementation and operation or one for all of them) for theimplementation and operation of identifying the assets per theguidelines, system requirements, system design, system build, andresulting system. The analysis system renders one or more secondevaluation ratings for the implementation and operation of protectingthe assets from issues per the guidelines, system requirements, systemdesign, system build, and resulting system.

The analysis system renders one or more third evaluation ratings for theimplementation and operation of detecting issues that may affect or areaffecting the assets per the guidelines, system requirements, systemdesign, system build, and resulting system. The analysis system rendersone or more fourth evaluation ratings for the implementation andoperation of responding to issues that may affect or are affecting theassets per the guidelines, system requirements, system design, systembuild, and resulting system.

The analysis system renders one or more fifth evaluation ratings for theimplementation and operation of recovering from issues that may affector are affecting the assets per the guidelines, system requirements,system design, system build, and resulting system. The analysis systemmay render an overall evaluation rating for the implementation and/orperformance based on the one or more first through one or more fifthevaluation ratings.

The analysis system 11 may further render an implementation and/oroperation evaluation rating regarding how well the discovered datacorrelates with the desired data. In other words, evaluate the levelimplementation and operation of the system. In this example, theanalysis system compares the disclosed data with the desired data. Ifthey substantially match, the implementation and/or operation of thesystem would receive a relatively high evaluation rating. The more thediscovered data differs from the desired data, the lower theimplementation and/or operation evaluation rating will be.

FIG. 55 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the system'sself-evaluation for identifying assets, protecting the assets fromissues, detecting issues that may affect or are affecting the assets,responding to issues that may affect or are affecting the assets, andrecovering from issues that affected the assets of a department based ondisclosed data and discovered data per the guidelines, systemrequirements, and system design.

For this specific example, the analysis system 10 obtains disclosed dataand discovered from the system regarding the guidelines, systemrequirements, and system design associated with the assets of thedepartment. From the disclosed data and discovered, the analysis systemrenders one or more first evaluation ratings (e.g., one for each ofguidelines, system requirements, and system design, or one for allthree) for the self-evaluation of identifying assets per the guidelines,system requirements, and system design. For instance, what resourcesdoes the system have with respect to its guidelines, systemrequirements, and/or system design for self-identifying of assets.

The analysis system renders one or more second evaluation ratings forthe self-evaluation of protecting the assets from issues per theguidelines, system requirements, and system design regarding. Theanalysis system renders one or more third evaluation ratings for theself-evaluation of detecting issues that may affect or are affecting theassets per the guidelines, system requirements, and system designregarding detection.

The analysis system renders one or more fourth evaluation ratings forthe self-evaluation of responding to issues that may affect or areaffecting the assets per the guidelines, system requirements, and systemdesign. The analysis system renders one or more fifth evaluation ratingsfor the self-evaluation of recovering from issues that affected theassets per the guidelines, system requirements, and system design. Theanalysis system may render an overall evaluation rating for theself-evaluation based on the one or more first through one or more fifthevaluation ratings.

FIG. 56 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating the understanding of theguidelines, system requirements, system design, system build, andresulting system for identifying assets, protecting the assets fromissues, detecting issues that may affect or are affecting the assets,responding to issues that may affect or are affecting the assets, andrecovering from issues that affected the assets of a department based ondisclosed data and discovered data.

For this specific example, the analysis system 10 obtains disclosed dataand discovered data from the system regarding guidelines, systemrequirements, system design, system build, and resulting systemassociated with the assets of the department. As a specific example, thedisclosed data includes guidelines that certain types of data shall beencrypted; a system requirement that specifies 128-bit AdvancedEncryption Standard (AES) for “y” types of documents; a system designthat includes 12 “x” type computers that are to be loaded with 128-bitAES software by company “M”, version 2.0 or newer; and a system buildand resulting system that includes 12 “x” type computers that have128-bit AES software by company “M”, version 2.1.

For this specific example, the discovered data includes the sameguideline as the disclosed data; a first system requirement thatspecifies 128-bit Advanced Encryption Standard (AES) for “y” types ofdocuments and a second system requirement that specifies 256-bitAdvanced Encryption Standard (AES) for “A” types of documents; a systemdesign that includes 12 “x” type computers that are to be loaded with128-bit AES software by company “M”, version 2.0 or newer, and 3 “z”type computers that are to be loaded with 256-bit AES software bycompany “N” version 3.0 or newer; and a system build and resultingsystem that includes 10 “x” type computers that have 128-bit AESsoftware by company “M” version 2.1, 2 “x” type computers that have128-bit AES software by company “M” version 1.3, 2 “z” type computersthat have 256-bit AES software by company “N” version 3.1, and 1 “z”type computer that has 256-bit AES software by company “K” version 0.1.

From just the disclosed data, the analysis system would render arelatively high evaluation rating for the understanding of theguidelines, system requirements, system design, system build, andresulting system associated with the assets of the department. Therelatively high evaluation rating would be warranted since the systembuild and resulting system included what was in the system design (e.g.,12 “x” type computers that have 128-bit AES software by company “M”,version 2.1). Further, the system design is consistent with the systemreequipments (e.g., 128-bit Advanced Encryption Standard (AES) for “y”types of documents), which is consistent with the guidelines (e.g.,certain types of data shall be encrypted).

From the discovered data, however, the analysis system would render arelatively low evaluation rating for the understanding of theguidelines, system requirements, system design, system build, andresulting system associated with the assets of the department. Therelatively low evaluation rating would be warranted since the systembuild and resulting system is not consistent with the system design(e.g., is missing 2 “x” type computers with the right encryptionsoftware, only has 2 “z” type computers with the right software, and hasa “z” type computer with the wrong software).

The analysis system would also process the evaluation ratings from thedisclosed data and from the discovered data to produce an overallevaluation rating for the understanding of the guidelines, systemrequirements, system design, system build, and resulting systemassociated with the assets of the department. In this instance, thedisclosed data does not substantially match the discovered data, whichindicates a lack of understanding of what's really in the system (i.e.,knowledge of the system). Further, since the evaluation rating from thediscovered data was low, the analysis system would produce a low overallevaluation rating for the understanding.

FIG. 57 is a diagram of an extension of the example of FIG. 56. In thisexample, the analysis system processes the data and/or evaluationratings to identify deficiencies and/or auto-corrections of at leastsome of the deficiencies. As shown, the disclosed data includes:

-   -   guidelines that certain types of data shall be encrypted;    -   a system requirement that specifies 128-bit Advanced Encryption        Standard (AES) for “y” types of documents;    -   a system design that includes 12 “x” type computers that are to        be loaded with 128-bit AES software by company “M”, version 2.0        or newer; and    -   a system build and resulting system that includes 12 “x” type        computers that have 128-bit AES software by company “M”, version        2.1.

As is also shown, the discovered data includes:

-   -   the same guideline as the disclosed data;    -   a first system requirement that specifies 128-bit Advanced        Encryption Standard (AES) for “y” types of documents and a        second system requirement that specifies 256-bit Advanced        Encryption Standard (AES) for “A” types of documents;    -   a system design that includes 12 “x” type computers that are to        be loaded with 128-bit AES software by company “M”, version 2.0        or newer, and 3 “z” type computers that are to be loaded with        256-bit AES software by company “N”, version 3.0 or newer; and    -   a system build and resulting system that includes:    -   10 “x” type computers that have 128-bit AES software by company        “M”, version 2.1;    -   2 “x” type computers that have 128-bit AES software by company        “M”, version 1.3;    -   2 “z” type computers that have 256-bit AES software by company        “N”, version 3.1; and    -   1 “z” type computer that has 256-bit AES software by company        “K”, version 0.1.

From this data, the analysis system identifies deficiencies 232 and,when possible, provides auto-corrections 235. For example, the analysissystem determines that the system requirements also included arequirement for 256-bit AES for “A” type documents. The analysis systemcan auto-correct this deficiency by updating the knowledge of the systemto include the missing requirement. This may include updating one ormore policies, one or more processes, one or more procedures, and/orupdating documentation.

As another example, the analysis system identifies the deficiency of thedesign further included 3 “z” type computers that are to be loaded with256-bit AES software by company “N”, version 3.0 or newer. The analysissystem can auto-correct this deficiency by updating the knowledge of thesystem to includes the thee “z” type computers with the correctsoftware. Again, this may include updating one or more policies, one ormore processes, one or more procedures, and/or updating documentation.

As another example, the analysis system identifies the deficiency of 2“x” type computers having old versions of the encryption software (e.g.,have version 1.3 of company M's 128-bit AES software instead of aversion 2.0 or newer). The analysis system can auto-correct thisdeficiency by updating the version of software for the two computers.

As another example, the analysis system identifies the deficiency of 1“z” type computer has the wrong encryption software (e.g., it hasversion 0.1 from company K and not version 3.0 or newer from company N).The analysis system can auto-correct this deficiency by replacing thewrong encryption software with the correct encryption software.

As another example, the analysis system identifies the deficiency of 1“z” type computer is missing from the system. The analysis system cannotauto-correct this deficiency since it is missing hardware. In thisinstance, the analysis system notifies a system admin of the missingcomputer.

FIG. 58 is a schematic block diagram of an embodiment of an evaluationprocessing module 254 that includes a plurality of comparators 360-362,a plurality of analyzers 363-365, and a deficiency correction module366. In general, the evaluation processing module 254 identifiesdeficiencies 232 and, when possible, determines auto-corrections 235from the ratings 219 and/or inputted data (e.g., disclosed data,discovered data, and/or desired data) based on evaluation parameters 266(e.g., disclosed to discovered deficiency criteria 368, discovered todesired deficiency criteria 370, disclosed to desired deficiencycriteria 372, disclosed to discovered compare criteria 373, discoveredto desired compare criteria 374, and disclosed to desired comparecriteria 375).

In an example, comparator 360 compares disclosed data and/or ratings 338and discovered data and/or ratings 339 based on the disclosed todiscovered compare criteria 373 to produce, if any, one or moredisclosed to discovered differences 367. As a more specific example, theanalysis system evaluates disclosed, discovered, and/or desired data toproduce one or more evaluation ratings regarding the understanding ofthe guidelines, system requirements, system design, system build, andresulting system associated with identifying the assets of thedepartment.

Each of the disclosed data, discovered data, and desired data includesdata regarding the guidelines, system requirements, system design,system build, and/or resulting system associated with identifying theassets of the department and/or the assets of the department. Recallthat disclosed data is the known data of the system at the outset of ananalysis, which is typically supplied by a system administrator and/oris obtained from data files of the system. The discovered data is thedata discovered about the system by the analysis system during theanalysis. The desired data is the data obtained by the analysis systemfrom system proficiency resources regarding desired guidelines, systemrequirements, system design, system build, and/or system operation.

For the understanding of the guidelines, system requirements, systemdesign, system build, and resulting system associated with identifyingthe assets of the department, the analysis system may produce one ormore evaluation ratings. For example, the analysis system produces anevaluation rating for:

-   -   understanding of the guidelines with respect to identifying        assets of the department from the disclosed data;    -   understanding of the guidelines with respect to identifying        assets of the department from the discovered data;    -   understanding of the guidelines with respect to identifying        assets of the department from the desired data;    -   understanding of the system requirements with respect to        identifying assets of the department from the disclosed data;    -   understanding of the system requirements with respect to        identifying assets of the department from the discovered data;    -   understanding of the system requirements with respect to        identifying assets of the department from the desired data;    -   understanding of the system design with respect to identifying        assets of the department from the disclosed data;    -   understanding of the system design with respect to identifying        assets of the department from the discovered data;    -   understanding of the system design with respect to identifying        assets of the department from the desired data;    -   understanding of the system build with respect to identifying        assets of the department from the disclosed data;    -   understanding of the system build with respect to identifying        assets of the department from the discovered data;    -   understanding of the system build with respect to identifying        assets of the department from the desired data;    -   understanding of the resulting system with respect to        identifying assets of the department from the disclosed data;    -   understanding of the resulting system with respect to        identifying assets of the department from the discovered data;    -   understanding of the resulting system with respect to        identifying assets of the department from the desired data;        and/or    -   an overall understanding of identifying the assets of the        department.

The disclosed to discovered compare criteria 373 specifies theevaluation ratings to be compared and/or which data of the discloseddata is to be compared to data of the discovered data. For example, thedisclosed to discovered compare criteria 373 indicates that the“understanding of the guidelines with respect to system design of thedepartment from the disclosed data” is to be compared to the“understanding of the system design with respect to identifying assetsof the department from the discovered data”. As another example, thedisclosed to discovered compare criteria 373 indicates that dataregarding system design of the disclosed data is to be compared with thedata regarding the system design of the discovered data.

In accordance with the disclosed to discovered compare criteria 373 andfor this specific example, the comparator 360 compares the“understanding of the guidelines with respect to system design of thedepartment from the disclosed data” with the “understanding of thesystem design with respect to identifying assets of the department fromthe discovered data” to produce, if any, one or more understandingdifferences. The comparator 360 also compares the data regarding systemdesign of the disclosed data with the data regarding the system designof the discovered data to produce, if any, one or more data differences.The comparator 360 outputs the one or more understanding differencesand/or the one or more data differences as the disclosed to discovereddifferences 367.

The analyzer 363 analyzes the disclosed to discovered differences 267 inaccordance with the disclosed to discovered deficiency criteria 368 todetermine whether a difference 267 constitutes a deficiency. If so, theanalyzer 363 includes it in the disclosed to discovered deficiencies232-1. The disclosed to discovered deficiency criteria 368 correspond tothe disclosed to discovered compare criteria 373 and specify how thedifferences 367 are to be analyzed to determine if they constitutedeficiencies 232-1.

As an example, the disclosed to discovered deficiency criteria 368specify a series of comparative thresholds based on the impact thedifferences have on the system. The range of impact is from none tosignificant with as many granular levels in between as desired. Fordifferences that have a significant impact on the system, thecomparative threshold is set to trigger a deficiency for virtually anydifference. For example, if the difference is regarding system security,then then threshold is set that any difference is a deficiency.

As another example, if the difference is regarding is inconsequentialinformation, then the threshold is set to not identify the difference asa deficiency. For example, the discovered data includes a PO date onNov. 2, 2020 for a specific purchase order and the disclosed data didn'tinclude a PO date, but the rest of the information regarding the PO isthe same for the disclosed and discovered data. In this instance, themissing PO date is inconsequential and would not be identified as adeficiency.

The deficiency correction module 366 receives the disclosed todiscovered deficiencies 232-1, if any, and determines whether one ormore of the deficiencies 232-1 can be auto-corrected to produce anauto-correction 235. In many instances, software deficiencies areauto-correctable (e.g., wrong software, missing software, out-of-datesoftware, etc.) while hardware deficiencies are not auto-correctable(e.g., wrong computing device, missing computing device, missing networkconnection, etc.).

The comparator 361 functions similarly to the comparator 360 to producediscovered to desired differences 369 based on the discovered dataand/or rating 339 and the desired data and/or rating 340 in accordancewith the discovered to desired compare criteria 374. The analyzer 364functions similarly to the analyzer 363 to produce discovered to desireddeficiencies 232-2 from the discovered to desired differences 369 inaccordance with the discovered to desired deficiency criteria 370. Thedeficiency correction module 366 auto-corrects, when possible, thediscovered to desired deficiencies 232-2 to produce auto-corrections235.

The comparator 362 functions similarly to the comparator 360 to producedisclosed to desired differences 371 based on the disclosed data and/orrating 338 and the desired data and/or rating 340 in accordance with thedisclosed to desired compare criteria 375. The analyzer 365 functionssimilarly to the analyzer 363 to produce disclosed to desireddeficiencies 232-3 from the disclosed to desired differences 371 inaccordance with the disclosed to desired deficiency criteria 372. Thedeficiency correction module 366 auto-corrects, when possible, thedisclosed to desired deficiencies 232-3 to produce auto-corrections 235.

While the examples were for the understanding of the system with respectto identifying assets of the department, the evaluation processingmodule 254 processes any combination of system aspects, evaluationaspects, and evaluation metrics in a similar manner. For example, theevaluation processing module 254 processes the implementation of thesystem with respect to identifying assets of the department to identifydeficiencies 232 and auto-corrections in the implementation. As anotherexample, the evaluation processing module 254 processes the operation ofthe system with respect to identifying assets of the department toidentify deficiencies 232 and auto-corrections in the operation of thesystem.

FIG. 59 is a state diagram of an example the analysis system analyzing asystem. From a start state 380, the analysis proceeds to anunderstanding of the system state 38) or to a test operations of theassets system functions, and/or security functions of a system state 386based on the desired analysis to be performed. For testing theunderstanding, the analysis proceeds to state 381 where theunderstanding of the assets, system functions, and/or security functionsof the system are evaluated. This may be done via documentation of thesystem, policies of the supported business, based upon a question andanswer session with personnel of the owner/operator of the system,and/or as discussed herein.

If the understanding of the system is inadequate, the analysis proceedsto the determine deficiencies in the understanding of the system state382. In this state 382, the deficiencies in understanding are determinedby processing differences and/or as discussed herein.

From state 382, corrections required in understanding the system areidentified and operation proceeds to state 383 in which a report isgenerated regarding understanding deficiencies and/or correctivemeasures to be taken. In addition, a report is generated and sent to theowner/operator of the other system. If there are no understandingdeficiencies and/or corrective measures, no auto correction is needed,and operations are complete at the done state.

If an autocorrect can be done, operation proceeds to state 384 where theanalysis system updates a determined ability to understand the othersystem. Corrections are then implemented, and operation proceeds back tostate 381. Note that corrections may be automatically performed for somedeficiencies but not others, depending upon the nature of thedeficiency.

From state 381, if the tested understanding of the system is adequate,operation proceeds to state 385 where a report is generated regarding anadequate understanding of the system and the report is sent. From state385 if operation is complete, operations proceed to the done state.Alternately, from state 385 operation may proceed to state 386 wheretesting of the assets, system functions and/or security functions of theother system is performed. If testing of the assets, system functions,and/or security functions of the system results in an adequate testresult, operation proceeds to state 390 where a report is generatedindicating adequate implementation and/or operation of the system andthe report is sent.

Alternately, at state 386 if the testing of the system results in aninadequate result, operations proceed to state 387 where deficiencies inthe assets, system functions, and/or security functions of the systemare tested. At state 387 differences are compared to identifydeficiencies in the assets, system functions, and/or security functions.The analysis then proceeds from state 387 to state 388 where a report isgenerated regarding corrective measures to be taken in response to theassets, system functions, and/or security functions deficiencies. Thereport is then sent to the owner/operator. If there are no deficienciesand/or corrective measures, no auto correction is needed, and operationsare complete at the done state. If autocorrect is required, operationproceeds to state 389 where the analysis system updates assets, systemfunctions, and/or security functions of the system. Corrections are thenimplemented and the analysis proceeds to state 386. Note thatcorrections may be automatically performed for some deficiencies but notothers, depending upon the nature of the deficiency.

FIG. 60 is a logic diagram of an example of an analysis system analyzinga system, or portion thereof. The method includes the analysis systemobtaining system proficiency understanding data regarding the assets ofthe system (step 400) and obtaining data regarding the owner/operator'sunderstanding of the assets (step 401). System proficiencies of step 400include industry best practices and regulatory requirements, forexample. The data obtained from the system at step 401 is based upondata received regarding the system or received by probing the system.

The data collected at steps 400 and 401 is then compared (step 402) anda determination is made regarding the comparison. If the comparison isfavorable, as determined at step 403, meaning that the systemproficiency understanding compares favorably to the data regardingunderstanding, operation is complete, a report is generated (step 412),and the report is sent (step 413). If the comparison is not favorable,as determined at step 403, operation continues with identifyingdeficiencies in the understanding of the system (step 404), identifyingcorrective measures (step 405), generating a corresponding report (step412) and sending the report (step 413).

The method also includes the analysis system obtaining systemproficiency understanding data of the system functions and/or securityimplementation and/or operation of the system (step 406) and obtainingdata regarding the owner/operator's understanding of the systemfunctions and/or security functions implementation and/or operation ofthe system (step 407). System proficiencies of step 406 include industrybest practices and regulatory requirements, for example. The dataobtained from the system at step 407 is based upon data receivedregarding the system or received by probing the system.

The data collected at steps 406 and 407 is then compared (step 414) anda determination is made regarding the comparison. If the comparison isfavorable, as determined at step 415, meaning that the systemproficiency understanding compares favorably to the data regardingunderstanding, operation is complete, a report is generated (step 412),and the report is sent (step 413). If the comparison is not favorable,as determined at step 415, operation continues with identifyingdeficiencies in the understanding of the system (step 416), identifyingcorrective measures (step 417), generating a corresponding report (step412) and sending the report (step 413).

The method further includes the analysis system comparing theunderstanding of the physical structure (obtained at step 401) with theunderstanding of the system functions and/or security functionsimplementation and/or operation (obtained at step 406) at step 408. Step408 essentially determines whether the understanding of the assetscorresponds with the understanding of the system functions and/orsecurity functions of the implementation and/or operation of the system.If the comparison is favorable, as determined at step 409, a report isgenerated (step 412), and the report is sent (step 413). If thecomparison is not favorable, as determined at step 409, the methodcontinues with identifying imbalances in the understanding (step 410),identifying corrective measures (step 410), generating a correspondingreport (step 412), and sending the report (step 413).

FIG. 61 is a logic diagram of another example of an analysis systemanalyzing a system, or portion thereof. The method begins at step 420where the analysis system determines a system evaluation mode (e.g.,assets, system functions, and/or security functions) for analysis. Themethod continues at step 421 where the analysis system determines asystem evaluation level (e.g., the system or a portion thereof). Forinstance, the analysis system identifies one or more system elements forevaluation.

The method continues at step 422 where the analysis system determines ananalysis perspective (e.g., understanding, implementation, operation,and/or self-evaluate). The method continues at step 423 where theanalysis system determines an analysis viewpoint (e.g., disclosed,discovered, and/or desired). The method continues at step 424 where theanalysis system determines a desired output (e.g., evaluation rating,deficiencies, and/or auto-corrections).

The method continues at step 425 where the analysis system determineswhat data to gather based on the preceding determinations. The methodcontinues at step 426 where the analysis system gathers data inaccordance with the determination made in step 425. The method continuesat step 427 where the analysis system determines whether the gathereddata is to be pre-processed.

If yes, the method continues at step 428 where the analysis systemdetermines data pre-processing functions (e.g., normalize, parse, tag,and/or de-duplicate). The method continues at step 429 where theanalysis system pre-processes the data based on the pre-processingfunctions to produce pre-processed data. Whether the data ispre-processed or not, the method continues at step 430 where theanalysis system determines one or more evaluation categories (e.g.,identify, protect, detect, respond, and/or recover) and/orsub-categories for evaluation. Note that this may be done prior to step425 and be part of determining the data to gather.

The method continues at step 431 where the analysis system analyzes thedata in accordance with the determine evaluation categories and inaccordance with a selected evaluation metric (e.g., process, policy,procedure, automation, certification, and/or documentation) to produceanalysis results. The method continues at step 432 where the analysissystem processes the analysis results to produce the desired output(e.g., evaluation rating, deficiencies, and/or auto-correct). The methodcontinues at step 432 where the analysis system determines whether toend the method or repeat it for another analysis of the system.

FIG. 62 is a logic diagram of another example of an analysis systemanalyzing a system or portion thereof. The method begins at step 440where the analysis system determines physical assets of the system, orportion thereof, to analyze (e.g., assets in the resulting system).Recall that a physical asset is a computing entity, a computing device,a user software application, a system software application (e.g.,operating system, etc.), a software tool, a network softwareapplication, a security software application, a system monitoringsoftware application, and the like.

The method continues at step 441 where the analysis system ascertainsimplementation of the system, or portion thereof (e.g., assets designedto be, and/or built, in the system). The method continues at step 442where the analysis system correlates components of the assets tocomponents of the implementation (e.g., do the assets of the actualsystem correlate with assets design/built to be in the system).

The method continues at step 443 where the analysis system scores thecomponents of the physical assets in accordance with the mappedcomponents of the implementation. For example, the analysis systemscores how well the assets of the actual system correlate with assetsdesign/built to be in the system. The scoring may be based on one ormore evaluation metrics (e.g. process, policy, procedure, automation,certification, and/or documentation). The method continues at step 444where the analysis system performs a function on the scores to obtain aresult (e.g., an evaluation rating, identified deficiencies, and/orauto-correction of deficiencies).

The method continues at step 445 where the analysis system determineswhether the result is equal or greater than a target result (e.g., theevaluation rating is a certain value). If yes, the method continues atstep 446 where the analysis system indicates that the system, or portionthereof, passes this particular test. If the results are less than thetarget result, the method continues at step 447 where the analysissystem identifies vulnerabilities in the physical assets and/or in theimplementation. For example, the analysis system determines that asecurity software application is missing from several computing devicesin the system, or portion thereof, being analyzed.

The method continues at step 448 where the analysis system determines,if possible, corrective measures of the identified vulnerabilities. Themethod continues at step 449 where the analysis system determineswhether the corrective measures can be done automatically. If not, themethod continues at step 451 where the analysis system reports thecorrective measures. If yes, the method continues at step 450 where theanalysis system auto-corrects the vulnerabilities.

FIG. 63 is a logic diagram of another example of an analysis systemanalyzing a system or portion thereof. The method begins at step 460where the analysis system determines physical assets of the system, orportion thereof, to analyze (e.g., assets and their intended operation).The method continues at step 461 where the analysis system ascertainsoperation of the system, or portion thereof (e.g., the operationsactually performed by the assets). The method continues at step 462where the analysis system correlates components of the assets tocomponents of operation (e.g., do the identified operations of theassets correlate with the operations actually performed by the assets).

The method continues at step 463 where the analysis system scores thecomponents of the physical assets in accordance with the mappedcomponents of the operation. For example, the analysis system scores howwell the identified operations of the assets correlate with operationsactually performed by the assets. The scoring may be based on one ormore evaluation metrics (e.g. process, policy, procedure, automation,certification, and/or documentation). The method continues at step 464where the analysis system performs a function on the scores to obtain aresult (e.g., an evaluation rating, identified deficiencies, and/orauto-correction of deficiencies).

The method continues at step 465 where the analysis system determineswhether the result is equal or greater than a target result (e.g., theevaluation rating is a certain value). If yes, the method continues atstep 466 where the analysis system indicates that the system, or portionthereof, passes this particular test. If the results are less than thetarget result, the method continues at step 467 where the analysissystem identifies vulnerabilities in the physical assets and/or in theoperation.

The method continues at step 468 where the analysis system determines,if possible, corrective measures of the identified vulnerabilities. Themethod continues at step 469 where the analysis system determineswhether the corrective measures can be done automatically. If not, themethod continues at step 471 where the analysis system reports thecorrective measures. If yes, the method continues at step 470 where theanalysis system auto-corrects the vulnerabilities.

FIG. 64 is a logic diagram of another example of an analysis systemanalyzing a system or portion thereof. The method begins at step 480where the analysis system determines system functions of the system, orportion thereof, to analyze. The method continues at step 481 where theanalysis system ascertains implementation of the system, or portionthereof (e.g., system functions designed to be, and/or built, in thesystem). The method continues at step 482 where the analysis systemcorrelates components of the system functions to components of theimplementation (e.g., do the system functions of the actual systemcorrelate with system functions design/built to be in the system).

The method continues at step 483 where the analysis system scores thecomponents of the system functions in accordance with the mappedcomponents of the implementation. For example, the analysis systemscores how well the system functions of the actual system correlate withsystem functions design/built to be in the system. The scoring may bebased on one or more evaluation metrics (e.g. process, policy,procedure, automation, certification, and/or documentation). The methodcontinues at step 484 where the analysis system performs a function onthe scores to obtain a result (e.g., an evaluation rating, identifieddeficiencies, and/or auto-correction of deficiencies).

The method continues at step 485 where the analysis system determineswhether the result is equal or greater than a target result (e.g., theevaluation rating is a certain value). If yes, the method continues atstep 486 where the analysis system indicates that the system, or portionthereof, passes this particular test. If the results are less than thetarget result, the method continues at step 487 where the analysissystem identifies vulnerabilities in the physical assets and/or in theimplementation.

The method continues at step 488 where the analysis system determines,if possible, corrective measures of the identified vulnerabilities. Themethod continues at step 489 where the analysis system determineswhether the corrective measures can be done automatically. If not, themethod continues at step 491 where the analysis system reports thecorrective measures. If yes, the method continues at step 490 where theanalysis system auto-corrects the vulnerabilities.

FIG. 65 is a logic diagram of another example of an analysis systemanalyzing a system or portion thereof. The method begins at step 500where the analysis system determines system functions of the system, orportion thereof, to analyze. The method continues at step 501 where theanalysis system ascertains operation of the system, or portion thereof(e.g., the operations associated with the system functions). The methodcontinues at step 502 where the analysis system correlates components ofthe system functions to components of operation (e.g., do the identifiedoperations of the system functions correlate with the operationsactually performed to provide the system functions).

The method continues at step 503 where the analysis system scores thecomponents of the system functions in accordance with the mappedcomponents of the operation. For example, the analysis system scores howwell the identified operations to support the system functions correlatewith operations actually performed to support the system functions. Thescoring may be based on one or more evaluation metrics (e.g. process,policy, procedure, automation, certification, and/or documentation). Themethod continues at step 504 where the analysis system performs afunction on the scores to obtain a result (e.g., an evaluation rating,identified deficiencies, and/or auto-correction of deficiencies).

The method continues at step 505 where the analysis system determineswhether the result is equal or greater than a target result (e.g., theevaluation rating is a certain value). If yes, the method continues atstep 506 where the analysis system indicates that the system, or portionthereof, passes this particular test. If the results are less than thetarget result, the method continues at step 507 where the analysissystem identifies vulnerabilities in the physical assets and/or in theoperation.

The method continues at step 508 where the analysis system determines,if possible, corrective measures of the identified vulnerabilities. Themethod continues at step 509 where the analysis system determineswhether the corrective measures can be done automatically. If not, themethod continues at step 511 where the analysis system reports thecorrective measures. If yes, the method continues at step 510 where theanalysis system auto-corrects the vulnerabilities.

FIG. 66 is a logic diagram of another example of an analysis systemanalyzing a system or portion thereof. The method begins at step 520where the analysis system determines security functions of the system,or portion thereof, to analyze. The method continues at step 521 wherethe analysis system ascertains implementation of the system, or portionthereof (e.g., security functions designed to be, and/or built, in thesystem). The method continues at step 522 where the analysis systemcorrelates components of the security functions to components of theimplementation (e.g., do the security functions of the actual systemcorrelate with security functions design/built to be in the system).

The method continues at step 523 where the analysis system scores thecomponents of the security functions in accordance with the mappedcomponents of the implementation. For example, the analysis systemscores how well the security functions of the actual system correlatewith security functions design/built to be in the system. The scoringmay be based on one or more evaluation metrics (e.g. process, policy,procedure, automation, certification, and/or documentation). The methodcontinues at step 524 where the analysis system performs a function onthe scores to obtain a result (e.g., an evaluation rating, identifieddeficiencies, and/or auto-correction of deficiencies).

The method continues at step 525 where the analysis system determineswhether the result is equal or greater than a target result (e.g., theevaluation rating is a certain value). If yes, the method continues atstep 526 where the analysis system indicates that the system, or portionthereof, passes this particular test. If the results are less than thetarget result, the method continues at step 527 where the analysissystem identifies vulnerabilities in the physical assets and/or in theimplementation.

The method continues at step 528 where the analysis system determines,if possible, corrective measures of the identified vulnerabilities. Themethod continues at step 529 where the analysis system determineswhether the corrective measures can be done automatically. If not, themethod continues at step 531 where the analysis system reports thecorrective measures. If yes, the method continues at step 530 where theanalysis system auto-corrects the vulnerabilities.

FIG. 67 is a logic diagram of another example of an analysis systemanalyzing a system or portion thereof. The method begins at step 540where the analysis system determines security functions of the system,or portion thereof, to analyze. The method continues at step 541 wherethe analysis system ascertains operation of the system, or portionthereof (e.g., the operations associated with the security functions).The method continues at step 542 where the analysis system correlatescomponents of the security functions to components of operation (e.g.,do the identified operations of the security functions correlate withthe operations actually performed to provide the security functions).

The method continues at step 543 where the analysis system scores thecomponents of the security functions in accordance with the mappedcomponents of the operation. For example, the analysis system scores howwell the identified operations to support the security functionscorrelate with operations actually performed to support the securityfunctions. The scoring may be based on one or more evaluation metrics(e.g. process, policy, procedure, automation, certification, and/ordocumentation). The method continues at step 544 where the analysissystem performs a function on the scores to obtain a result (e.g., anevaluation rating, identified deficiencies, and/or auto-correction ofdeficiencies).

The method continues at step 545 where the analysis system determineswhether the result is equal or greater than a target result (e.g., theevaluation rating is a certain value). If yes, the method continues atstep 546 where the analysis system indicates that the system, or portionthereof, passes this particular test. If the results are less than thetarget result, the method continues at step 547 where the analysissystem identifies vulnerabilities in the physical assets and/or in theoperation.

The method continues at step 548 where the analysis system determines,if possible, corrective measures of the identified vulnerabilities. Themethod continues at step 549 where the analysis system determineswhether the corrective measures can be done automatically. If not, themethod continues at step 551 where the analysis system reports thecorrective measures. If yes, the method continues at step 550 where theanalysis system auto-corrects the vulnerabilities.

FIG. 68 is a logic diagram of an example of an analysis systemdetermining an issue detection rating for a system, or portion thereof.Issue detection rating is also referred to herein as detect ratingand/or detection rating. An issue includes an event, incident, alert,anomalous activity and/or behavior, and any other system occurrence ofinterest. The method begins at step 560 where the analysis systemdetermines a system aspect (see FIG. 69) of a system for an issuedetection evaluation. An issue detection evaluation includes evaluatingthe system aspect's anomalies and event awareness, the system aspect'scontinuous security monitoring, and/or the system aspect's issuedetection process analysis.

The method continues at step 561 where the analysis system determines atleast one evaluation perspective for use in performing the issuedetection evaluation on the system aspect. An evaluation perspective isan understanding perspective, an implementation perspective, anoperation (e.g., an operational performance) perspective, or aself-analysis perspective. An understanding perspective is with regardto how well the assets, system functions, and/or security functions areunderstood. An implementation perspective is with regard to how well theassets, system functions, and/or security functions are implemented. Anoperation perspective (also referred to herein as a performanceperspective) is with regard to how well the assets, system functions,and/or security functions operate. A self-analysis (or self-evaluation)perspective is with regard to how well the system self-evaluates theunderstanding, implementation, and/or operation of assets, systemfunctions, and/or security functions.

The method continues at step 562 where the analysis system determines atleast one evaluation viewpoint for use in performing the issue detectionevaluation on the system aspect. An evaluation viewpoint is a disclosedviewpoint, a discovered viewpoint, or a desired viewpoint. A disclosedviewpoint is with regard to analyzing the system aspect based on thedisclosed data. A discovered viewpoint is with regard to analyzing thesystem aspect based on the discovered data. A desired viewpoint is withregard to analyzing the system aspect based on the desired data.

The method continues at step 563 where the analysis system obtains issuedetection data regarding the system aspect in accordance with the atleast one evaluation perspective and the at least one evaluationviewpoint. Issue detection data is data obtained about the systemaspect. The obtaining of issue detection data will be discussed ingreater detail with reference to FIGS. 72-79.

The method continues at step 564 where the analysis system calculates anissue detection rating as a measure of system issue detection maturityfor the system aspect based on the issue detection data, the at leastone evaluation perspective, the at least one evaluation viewpoint, andat least one evaluation rating metric. An evaluation rating metric is aprocess rating metric, a policy rating metric, a procedure ratingmetric, a certification rating, a documentation rating metric, or anautomation rating metric. The calculating of an issue detection ratingwill be discussed in greater detail with reference to FIGS. 80-108. Asused herein, maturity refers to level of development, level of operationreliability, level of operation predictability, level of operationrepeatability, level of understanding, level of implementation, level ofadvanced technologies, level of operation efficiency, level ofproficiency, and/or state-of-the-art level.

FIG. 69 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereofin particular, for determining a system aspect. The method begins atstep 565 where the analysis system determines at least one systemelement of the system. A system element includes one or more systemassets which is a physical asset and/or a conceptual asset. For example,a physical asset is a computing entity, a computing device, a usersoftware application, a system software application (e.g., operatingsystem, etc.), a software tool, a network software application, asecurity software application, a system monitoring software application,and the like. A system element includes an organization identifier, adivision identifier, a department identifier, a group identifier, asub-group identifier, a device identifier, a software identifier, and/oran internet protocol address identifier.

The method continues at step 566 where the analysis system determines atleast one system criteria of the system. A system criteria is systemguidelines, system requirements, system design, system build, orresulting system. Evaluation based on system criteria assists withdetermining where a deficiency originated and/or how it might becorrected. For example, if the system requirements were lacking arequirement for detecting a particular type of threat, the lack ofsystem requirements could be identified and corrected.

The method continues at step 567 where the analysis system determines atleast one system mode of the system. A system mode is assets, systemfunctions, or system security. The method continues at step 568 wherethe analysis system determines the system aspect based on the at leastone system element, the at least one system criteria, and the at leastone system mode. As an example, a system aspect is determined to be forassets with respect to system requirements of system elements in aparticular division. As another example, a system aspect is determinedto be for system functions with respect to system design and/or systembuild of system elements in a particular division. As yet anotherexample, a system aspect is determined to be for assets, systemfunctions, and security functions with respect to guidelines, systemrequirements, system design, system build, and resulting system ofsystem elements in the organization (e.g., the entiresystem/enterprise).

FIG. 70 is a schematic block diagram of an example of an analysis system10 determining an issue detection rating for a system, or portionthereof. In this example, the control module 256 receives an input 271from the system user interface module 81 loaded on a system 11. Theinput 271 identifies the system aspect to be analyzed and how it is tobe analyzed.

The control module 256 determines one or more system elements, one ormore system criteria, and one or more system modes based on the systemaspect. The control module 256 also determines one or more evaluationperspectives, one or more evaluation viewpoints, and/or one or moreevaluation rating metrics from the input. As an example, the input 271could specify the evaluation perspective(s), the evaluationviewpoint(s), the evaluation rating metric(s), and/or analysisoutput(s). As another example, the input 271 indicates a desiredanalysis output (e.g., an evaluation rating, deficiencies identified,and/or deficiencies auto-corrected). From this input, the control module256 determines the evaluation perspective(s), the evaluationviewpoint(s), the evaluation rating metric(s) to fulfill the desiredanalysis output.

In addition, the control module 256 generates data gathering parameters263, pre-processing parameters 264, data analysis parameters 265, and/orevaluation parameters 266 as discussed with reference to FIG. 35. Thedata input module 250 obtains issue detection data in accordance withthe data gathering parameters 263 from the data extraction module(s) 80loaded on the system 11, other external feeds 258, and/or systemproficiency data 260.

The pre-processing module 251 processes the issue detection data inaccordance with the pre-processing parameters 264 to producepre-processed data 414. The issue detection data and/or thepre-processed data 414 may be stored in the database 275. The dataanalysis module 252 calculates an issue detection rating 219 based onthe pre-processed data 414 in accordance with the data analysisparameters 265 and the analysis modeling 268.

If the requested analysis output was for an evaluation rating only, thedata output module 255 outputs the issue detection rating 219 as theoutput 269. The system user interface module 80 renders a graphicalrepresentation of the issue detection rating and the database 275 storesit.

If the required analysis output included issue detection deficiencies,then the evaluation processing module 254 evaluates the issue detectionrating 219 and may further evaluate the pre-processed data to identifyone or more deficiencies 232. In addition, the evaluation processingmodule 254 determines whether a deficiency can be auto-corrected and, ifso, determines the auto-correction 235. In this instance, the dataoutput module 255 outputs the issue detection rating 219, thedeficiencies 232, and the auto-corrections 235 as output 269 to thedatabase 275, the system user interface module 81, and the remediationmodule 257.

The system user interface module 80 renders a graphical representationof the issue detection rating, the deficiencies, and/or theauto-corrections. The database 275 stores the issue detection rating,the deficiencies, and/or the auto-corrections. The remediation module257 processes the auto-corrections 235 within the system 11, verifiesthe auto-corrections, and then records the execution of theauto-correction and its verification.

FIG. 71 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating, with respect to process,policy, procedure, certification, documentation, and/or automation, theunderstanding of the system build for issue detection (e.g., the“detect” evaluation category) security functions of an organization(e.g., the entire enterprise) based on disclosed data to produce anevaluation rating.

For this specific example, the analysis system 10 obtains disclosed datafrom the system regarding the system build associated with the issuedetection security functions of the organization. From the discloseddata, the analysis system renders a first evaluation rating for theunderstanding of the system build for issue detection security functionswith respect to an evaluation rating metric of process. The analysissystem renders a second evaluation rating for the understanding of thesystem build for issue detection security functions with respect to anevaluation rating metric of policy. The analysis system renders a thirdevaluation rating for the understanding of the system build for issuedetection security functions with respect to an evaluation rating metricof procedure. The analysis system renders a fourth evaluation rating forthe understanding of the system build for issue detection securityfunctions with respect to an evaluation rating metric of certification.The analysis system renders a fifth evaluation rating for theunderstanding of the system build for issue detection security functionswith respect to an evaluation rating metric of documentation. Theanalysis system renders a sixth evaluation rating for the understandingof the system build for issue detection security functions with respectto an evaluation rating metric of automation.

The analysis system 11 generates the issue detection rating for theunderstanding of the system build for issue detection security functionsbased on the six evaluation ratings. As example, each of the sixevaluation rating metrics has a maximum potential rating (e.g., 50 forprocess, 20 for policy, 15 for procedure, 10 for certification, 20 fordocumentation, and 20 for automation), which has a maximum rating of135. Continuing with this example, the first evaluation rating based onprocess is 35; the second evaluation rating based on policy is 10; thethird evaluation rating based on procedure is 10; the fourth evaluationrating based on certification is 10; the fifth evaluation rating basedon documentation is 15; and the sixth evaluation rating based onautomation is 20, resulting in a cumulative score of 100 out of apossible 135. This rating indicates that there is room for improvementand provides a basis for identifying deficiencies.

FIG. 72 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular, for obtaining issue detection data, which is a collectionof issue detection information. The method begins at step 570 where theanalysis system determines data gathering parameters regarding thesystem aspect in accordance with the at least one evaluationperspective, the at least one evaluation viewpoint, and the least oneevaluation rating metric. The generation of data gathering parameterswill be discussed in greater detail with reference to FIG. 74.

The method continues at step 571 where the analysis system identifiessystem elements of the system aspect based on the data gatheringparameters and obtains issue detection information from the systemelements in accordance with the data gathering parameters. The obtainingof the issue detection information is discussed in greater detail withreference to FIG. 73.

The method continues at step 572 where the analysis system records theissue detection information from the system elements to produce theissue detection data. As an example, the analysis system stores theissue detection information in the database. As another example, theanalysis system temporarily stores the issue detection information inthe data input module. As yet another example, the analysis system usessome form of retaining a record of the issue detection information.Examples of issue detection information are provided with reference toFIGS. 75-79.

FIG. 73 is a logic diagram of an analysis system determining an issuedetection rating for a system, or portion thereof; in particular, forobtaining the issue detection information. The method begins at step 573where the analysis system probes (e.g., push and/or pull informationrequests) a system element in accordance with the data gatheringparameters to obtain a system element data response. The analysis systemwould do this for most, if not all of the system elements of the systemaspect (e.g., the system, or portion of the system, being evaluated).

The method continues at step 574 where the analysis system identifiesvendor information from the system element data response. For example,vendor information includes vendor name, a model name, a product name, aserial number, a purchase date, and/or other information to identify thesystem element. The method continues at step 575 where the analysissystem tags the system element data response with the vendorinformation.

FIG. 74 is a logic diagram of a further of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular, for determining the data gathering parameters for theevaluation category of issue detection (i.e., detect). The method beginsat step 576 where the analysis system, for the system aspect, ascertainsidentity of one or more system elements of the system aspect. For asystem element of the system aspect, the method continues at step 577where the analysis system determines a first data gathering parameterbased on at least one system criteria (e.g., guidelines, systemrequirements, system design, system build, and/or resulting system) ofthe system aspect. For example, if the determined selected criteria issystem requirements, then the first data gathering parameter would be tosearch for system requirement information.

The method continues at step 578 where the analysis system determines asecond data gathering parameter based on at least one system mode (e.g.,assets, system functions, and/or security functions). For example, ifthe determined selected mode is system functions, then the second datagathering parameter would be to search for system function information.

The method continues at step 579 where the analysis system determines athird data gathering parameter based on the at least one evaluationperspective (e.g., understanding, implementation, operation, and/orself-evaluation). For example, if the determined selected evaluationperspective is operation, then the third data gathering parameter wouldbe to search for information regarding operation of the system aspect.

The method continues at step 580 where the analysis system determines afourth data gathering parameter based on the at least one evaluationviewpoint (e.g., disclosed data, discovered data, and/or desired data).For example, if the determined selected evaluation viewpoint isdisclosed and discovered data, then the fourth data gathering parameterwould be to obtain for disclosed data and to obtain discovered data.

The method continues at step 581 where the analysis system determines afifth data gathering parameter based on the at least one evaluationrating metric (e.g., process, policy, procedure, certification,documentation, and/or automation). For example, if the determinedselected evaluation rating metric is process, policy, procedure,certification, documentation, and automation, then the fifth datagathering parameter would be to search for data regarding process,policy, procedure, certification, documentation, and automation.

The analysis system generates the data gathering parameters from thefirst through fifth data gathering parameters. For example, the datagathering parameters include search for information regarding processes,policies, procedures, certifications, documentation, and/or automation(fifth parameter) pertaining to issue detection (selected evaluationcategory) system requirements (first parameter) for system operation(third parameter) of system functions (second parameter) from disclosedand discovered data (fourth parameter).

FIG. 75 is a diagram of an example of issue detection data for use by ananalysis system to generate an issue detection rating for a system, orportion thereof. For the evaluation category of issue detection (i.e.,detect), the sub-categories and/or sub-sub categories are cues fordetermining what data to gather for an issue detection evaluation. Thesub-categories include anomalies and event awareness, continuoussecurity monitoring, and issue detection processes analysis. As such,the issue detection data may include a collection of anomalies and eventawareness information, continuous security monitoring information, andissue detection processes analysis information.

The anomalies and event awareness sub-category includes the sub-subcategories of baseline of network operations and expected data flowsestablishment and management, detected event analysis, event dataaggregation and correlation, event impact determination, and incidentalert thresholds establishment. The anomalies and event awarenessinformation includes information regarding how these sub-sub categoriesare implemented (e.g., understood to be implemented, actuallyimplemented, etc.).

The continuous security monitoring sub-category includes the sub-subcategories of network monitoring, physical environment monitoring,personnel activity monitoring, malicious code detection, unauthorizedmobile code detection, external service provider activity monitoring,monitoring for unauthorized personnel, monitoring for unauthorizedconnections, monitoring for unauthorized devices, monitoring forunauthorized software, and vulnerability scanning. The continuoussecurity monitoring information includes information regarding how thesesub-sub categories are implemented (e.g., understood to be implemented,actually implemented, etc.).

The issue detection processes analysis sub-category includes the sub-subcategories of roles and responsibilities for issue detection aredefined, issue detection activities comply with all applicablerequirements (i.e., compliance verification), issue detection processtesting, event detection information communication, and issue detectionprocesses improvement. The issue detection processes analysisinformation includes information regarding how these sub-sub categoriesare implemented (e.g., understood to be implemented, actuallyimplemented, etc.).

FIG. 76 is a diagram of another example of issue detection data for useby an analysis system to generate an issue detection rating for asystem, or portion thereof. In this example, the issue detection dataincludes one or more diagrams, one or more reports, one or more logs,one or more application information records, one or more userinformation records, one or more device information records, one or moresystem tool records, one or more system plans, and/or one or more otherdocuments regarding a system aspect.

A diagram is a data flow diagram, an HLD diagram, an LLD diagram, a DLDdiagram, an operation flowchart, a data flow diagram, a securityoperations center (SOC) processes diagram, a software architecturediagram, a hardware architecture diagram, and/or other diagram regardingthe design, build, and/or operation of the system, or a portion thereof.A report may be a compliance report, a security report (e.g., riskassessment, user behavioral analysis, exchange traffic and use, etc.), asales report, a trend report, an inventory summary, or any kind ofreport generated by an aspect of the system and/or an outside sourceregarding the aspect of the system. For example, a compliance report isa report created to ensure system compliance with industry standards,laws, rules, and regulations set by government agencies and regulatorybodies. A compliance report may include information to ensure compliancewith the Payment Card Industry Data Security Standard (PCI-DSS), HealthInsurance Portability and Accountability Act (HIPPA), etc. A securityreport summarizes a system's weaknesses based on security scan resultsand other analytic tools.

Logs include machine data generated by applications or theinfrastructure used to run applications. Logs create a record of eventsthat happen on system components such as network devices (e.g.,firewall, router, switch, load balancer, etc.), user applications,databases, servers, and operating systems. An event is a change in thenormal behavior of a given system, process, environment, or workflow.Events can be positive, neutral, or negative. An average organizationexperiences thousands of events every day (e.g., an email, update tofirewalls, etc.). Logs include log files, event logs, transaction logs,message logs, etc. For example, a log file is a file that records eventsthat occur in an operating system, other software, or between differentusers of communication software. A transaction log is a file ofcommunications between a system and the users of that system, or a datacollection method that automatically captures the type, content, or timetransactions made by a person from a terminal with that system. Asanother example, an event log provides information about networktraffic, usage, and other conditions. For example, an event log maycapture all logon sessions to a network, along with account lockouts,failed password attempts, application events, etc. An event log storesthis data for retrieval by security professionals or automated securitysystems.

An application information record is a record regarding one or moreapplications of the system. For example, an application informationrecord includes a list of user applications and/or a list of systemapplications (e.g., operating systems) of the system or a portionthereof. A list of applications includes vendor information (e.g., name,address, contact person information, etc.), a serial number, a softwaredescription, a software model number, a version, a generation, apurchase date, an installation date, a service date, use and/or userinformation, and/or other mechanism for identifying applications.

A user information record is a record regarding one or more users of thesystem. For example, a user information record may include one or morelists of users and affiliations of users with the system, or portionthereof. As another example, a user information record include a log ofuse of the one or more assets by a user or others. A user informationrecord may also include privileges and/or restrictions imposed on theuse of the one or more assets (e.g., access control lists). A userinformation record may include roles and responsibilities of personnel(e.g., from a personnel handbook, access control list, data flowinformation, human resources documentation, and/or other systeminformation).

A device information record is a record regarding one or more devices ofthe system. For example, a device information record may include a listof network devices (e.g., hardware and/or software), a list of userdevices (e.g., hardware and/or software), a list of security devices(e.g., hardware and/or software), a list of servers (e.g., hardwareand/or software), and/or a list of any other devices of the system or aportion thereof. Each list includes device information pertaining to thedevices in the list.

The device information may include vendor information (e.g., name,address, contact person information, etc.), a serial number, a devicedescription, a device model number, a version, a generation, a purchasedate, an installation date, a service date, and/or other mechanism foridentifying a device. The device information may also include purchases,installation notes, maintenance records, data use information, and ageinformation. A purchase is a purchase order, a purchase fulfillmentdocument, bill of laden, a quote, a receipt, and/or other informationregarding purchases of assets of the system, or a portion thereof. Aninstallation note is a record regarding the installation of an asset ofthe system, or portion thereof. A maintenance record is a recordregarding each maintenance service performed on an asset of the system,or portion thereof.

As an example, a list of user devices and associated user deviceinformation may include (e.g., in tabular form) user ID, user level,user role, hardware (HW) information, IP address, user applicationsoftware (SW) information, device application software (SW) information,device use information, and/or device maintenance information. A user IDmay include an individual identifier of a user and may further includean organization ID, a division ID, a department ID, a group ID, and/or asub-group ID associated with the user.

A user level (e.g., C-Level, director level, general level) includesoptions for data access privileges, data access restrictions, networkaccess privileges, network access restrictions, server accessprivileges, server access restrictions, storage access privileges,storage access restrictions, required user applications, required deviceapplications, and/or prohibited user applications.

A user role (e.g., project manager, engineer, quality control,administration) includes further options for data access privileges,data access restrictions, network access privileges, network accessrestrictions, server access privileges, server access restrictions,storage access privileges, storage access restrictions, required userapplications, required device applications, and/or prohibited userapplications.

The HW information field may store information regarding the hardware ofthe device. For example, the HW information includes informationregarding a computing device such as vendor information, a serialnumber, a description of the computing device, a computing device modelnumber, a version of the computing device, a generation of the computingdevice, and/or other mechanism for identifying a computing device. TheHW information may further store information regarding the components ofthe computing device such as the motherboard, the processor, videographics card, network card, connection ports, and/or memory.

The user application SW information field may store informationregarding the user applications installed on the user's computingdevice. For example, the user application SW information includesinformation regarding a SW program (e.g., spreadsheet, word processing,database, email, etc.) such as vendor information, a serial number, adescription of the program, a program model number, a version of theprogram, a generation of the program, and/or other mechanism foridentifying a program. The device SW information may include similarinformation, but for device applications (e.g., operating system,drivers, security, etc.).

The device use data field may store data regarding the use of the device(e.g., use of the computing device and software running on it). Forexample, the device use data includes a log of use of a userapplication, or program (e.g., time of day, duration of use, dateinformation, etc.). As another example, the device use data includes alog of data communications to and from the device. As yet anotherexample, the device use data includes a log of network accesses. As afurther example, the device use data includes a log of server access(e.g., local and/or remote servers). As still further example, thedevice use data includes a log of storage access (e.g., local and/orremote memory).

The maintenance field stores data regarding the maintenance of thedevice and/or its components. As an example, the maintenance dataincludes a purchase date, purchase information, an installation date,installation notes, a service date, services notes, and/or othermaintenance data of the device and/or its components.

Note that, when device information records include applicationinformation, the application information records may include at least aportion of the device information records and vice versa.

A system tool record is a record regarding one or more tools of thesystem. For example, a system tool record includes a list of systemtools of the system or a portion thereof and information related to thesystem tools. System tools may include one or more security tools, oneor more data segmentation tools, one or more boundary detection tools,one or more data protection tools, one or more infrastructure managementtools, one or more encryption tools, one or more exploit protectiontools, one or more malware protection tools, one or more identitymanagement tools, one or more access management tools, one or moresystem monitoring tools, one or more verification tools, and/or one ormore vulnerability management tools.

A system tool may also be an infrastructure management tool, a networkmonitoring tool, a network strategy and planning tool, a networkmanaging tool, a Simple Network Management Protocol (SNMP) tool, atelephony monitoring tool, a firewall monitoring tool, a bandwidthmonitoring tool, an IT asset inventory management tool, a networkdiscovery tool, a network asset discovery tool, a software discoverytool, a security discovery tool, an infrastructure discovery tool,Security Information & Event Management (SIEM) tool, a data crawlertool, and/or other type of tool to assist in discovery of assets,functions, security issues, implementation of the system, and/oroperation of the system.

System tool information may include vendor information (e.g., name,address, contact person information, etc.), a serial number, a systemtool description, a system tool model number, a version, a generation, apurchase date, an installation date, a service date, and/or othermechanism for identifying a system tool. Note that the system tool maybe implemented by one or more devices and/or applications included inone or more of the device information records and the applicationinformation records such that some overlap in system tool records,device information records, and application information records mayoccur.

System plans include business plans, operational plans, system securityplans, system design specifications, etc. For example, a system securityplan (SSP) is a document that identifies the functions and features of asystem, including all its hardware and software installed on the system.System design specifications may include security specifications,hardware specifications, software specifications, data flowspecifications, business operation specifications, build specifications,and/or other specifications regarding the system, or a portion thereof.

Other documents may include internal documents such as memos, calendarentries, notes, emails, training materials, etc. Training materials mayinclude at least a portion of a personnel handbook, presentation slidesfrom a cybersecurity training session, links to videos and/or audiotutorials, handouts from a cybersecurity training session, user manualsregarding security devices and/or tools, training email communications,etc.

FIG. 77 is another example of issue detection data for use by ananalysis system to generate an issue detection rating for a system, orportion thereof. In this example, disclosed anomalies and eventawareness information is gathered from various disclosed data sources ofan organization as a portion of the disclosed issue detectioninformation collected as issue detection data.

As shown, disclosed organization diagrams relevant to anomalies andevent awareness include a network operations diagram and expected dataflow diagrams for users and systems of the organization. Disclosedsystem tools relevant to anomalies and event awareness include a networkmonitoring tool and a log management tool. For example, the networkmonitoring tool may be a simple network management protocol (SNMP). InSNMP, monitored devices are installed with an agent software and anetwork management system monitors each device and communicatesinformation from those devices to an administrator. As another example,the network monitoring tool may include NetFlow monitoring. NetFlow is astandard for collecting network traffic statistics (e.g., IP addresses,ports, protocol, timestamp, number of bytes, packets, flags, etc.) fromrouters, switches, specialized network probes, and/or other networkdevices. With NetFlow monitoring, security teams can detect changes innetwork behavior to identify anomalies indicative of a security breach.

A log management tool generally performs a variety of functions such aslog collection, central aggregation, storage and retention of logs, logrotation, analysis, and log reporting. The log management tool may be aSecurity Information & Event Management (SIEM) tool. A SIEM tool allowsan organization to monitor network activity and implements a variety ofsecurity techniques such as log management, security event management,security information management, and security event correlation.

Disclosed device information and application information relevant toanomalies and event awareness include a list of security devices(software and/or hardware), a list of network devices (software and/orhardware), a list of system applications (e.g., operating systems), alist of user applications, a list of servers (software and/or hardware),and a list of storage devices (software and/or hardware).

One or more of the disclosed devices or applications may be associatedwith a system security tool. For example, security devices may includeone or more log forwarding devices and a centralized log server requiredfor a log management tool to collect and store data from varioussources. As another example, a network packet collector is a securitydevice required to collect network traffic for input to the networkmonitoring tool. An entire detailed accounting of organization networkdevices, applications, operating systems, servers, databases, etc., isrelevant to anomalies and event awareness because these items generatelogs and/or are monitored for anomalous activity.

Portions of the disclosed system security plan relevant to anomalies andevent awareness include security analyst team roles and responsibilitiesas well as any specific processes, procedures, plans, data flows,policies, and requirements related to detecting anomalies and events.The security analyst team roles and responsibilities may include day today activities of each team member, verification processes, informationsharing requirements, hierarchy of roles, user privileges, and reportingrequirements.

In this example, the disclosed system security plan discloses a networkmonitoring process, a log management process, and event impact andincident alert processes. A network monitoring process may include arequirement to send network monitoring alerts generated by the networkmonitoring tool to a security analyst team for further analysis andremediation. Similarly, a log management process may include arequirement to send alerts generated by the log management tool to asecurity analyst team for further analysis and remediation. Event impactand incident alert processes may include log analysis procedures todetermine event impact and methods to establish appropriate incidentalert settings. An incident is a change in a system that negativelyimpacts the organization. For example, an incident might take place whena cyberattack occurs. An incident alert is a notification of acybersecurity incident.

Disclosed reports relevant to anomalies and event awareness includesecurity reports generated by one or more of the log management tool,the network monitoring tool, and the security analyst team.Additionally, compliance reports may contain relevant information toanomalies and event awareness such as previous security issues andapplication of remedies. Numerous other pieces of disclosed anomaliesand event awareness information may be gathered from various informationsources of the organization. The above examples are far from exhaustive.

FIG. 78 is another example of issue detection data for use by ananalysis system to generate an issue detection rating for a system, orportion thereof. In this example, disclosed continuous securitymonitoring information is gathered from various disclosed data sourcesof an organization as a portion of the disclosed issue detectioninformation recorded as issue detection data.

As shown, one or more portions of a disclosed system security planrelevant to continuous security monitoring include specific processes(e.g., network monitoring processes), procedures, plans, data flows,policies (e.g., physical environment threat protections), andrequirements related to continuous security monitoring. As an example,the system security plan includes policies that all user devices musthave antivirus software installed, that the antivirus software becentrally managed, and that unknown networks are prohibited. As anotherexample, a network monitoring process may include a requirement to sendalerts from a network monitoring tool to a security analyst team forfurther analysis and remediation. As another example, physicalenvironment threat protections may include policies against eating ordrinking around equipment, fire hazard avoidance policies, maintenanceof building security and environmental threat alarms, etc.

Disclosed device information relevant to continuous security monitoringinclude a list of security devices such as firewalls and intrusiondetection systems, antivirus software and associated devices, malwareprotection software and associated devices, automatic update andautomatic installation features of required software, and on-site alarmsystems. Disclosed system tools relevant to continuous securitymonitoring include a network monitoring tool and any tools pertaining touse and deployment of malware protection software. Other examples ofcontinuous security monitoring tools not disclosed here may include anetwork behavior anomaly detection (NBAD) tool for personnel and networkmonitoring, a cloud access security broker (CASB) tool for externalservice provider activity monitoring, etc. NBAD continuously tracksnetwork characteristics such as traffic volume, bandwidth use, andprotocol use in real time and generates an alarm if a strange event ortrend is detected indicative of a potential security threat. CASB is anon-premises or cloud-hosted software placed between cloud serviceconsumers and cloud server providers to enforce security, compliance,and governance policies for cloud applications.

Disclosed training materials relevant to continuous security monitoringinclude documents related to phishing attack awareness training, emailand website security training materials, physical environment securitytraining materials, dates and times of cybersecurity training sessions,and the participants involved in the disclosed cybersecurity trainingsessions.

Disclosed user information and application information relevant tocontinuous security monitoring may include user affiliations withdevices (hardware and/or software), access control lists, and personnelroles. For example, based on a user's role within the organization, theuser may have certain software requirements, access privileges, accessrestrictions, etc. Numerous other pieces of disclosed continuoussecurity monitoring information may be gathered from various informationsources of the organization. The above examples are far from exhaustive.

FIG. 79 is another example of issue detection data for use by ananalysis system to generate an issue detection rating for a system, orportion thereof. In this example, disclosed issue detection processesanalysis information is gathered from various disclosed data sources ofan organization as a portion of the disclosed issue detectioninformation recorded as issue detection data.

As shown, portions of the system security plan relevant to issuedetection processes analysis include specific processes, procedures,plans, data flows, policies, and requirements generally or specificallyrelated to issue detection. As an example, the system security planincludes documented issue detection policies and processes and issuedetection requirements (e.g., based on compliance requirements). Systemdiagrams relevant to issue detection processes analysis include issuedetection flowcharts that visually illustrate processes described in thesystem security plan. System tools relevant to issue detection processesinclude verification tools for testing and improving the issue detectionprocesses of the system.

Portions of personnel handbooks and/or human resources (HR) documentsrelevant to issue detection processes analysis include the roles andresponsibilities of those involves in issued detection processes and anypersonnel guidelines. For example, the portions of personnel handbooksand/or HR documents includes the roles and responsibilities of the ITdepartment, the roles and responsibilities of the security analyst team,the roles and responsibilities of the legal department, and issuedetection process guidelines.

Disclosed training materials relevant to issue detection processesanalysis include all issue detection training materials, dates and timesof cybersecurity training sessions, and the participants involved in thedisclosed cybersecurity training sessions.

Disclosed reports and internal documents relevant to issue detectionprocesses include compliance reports and personnel communications.Compliance reports include details regarding issue detection processesanalysis conducted and the results of such analysis. Personnelcommunications may include any relevant issue detection process analysisinformation. For example, an issue detection process may specify thatthe security analyst team must notify legal of incidents. An emailcommunication from the security analyst team to the legal departmentregarding an incident is a disclosed personnel communication thatdemonstrates implementation of this process. Numerous other pieces ofdisclosed issue detection processes analysis information may be gatheredfrom various information sources of the organization. The above examplesare far from exhaustive.

FIG. 80 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular, for calculating the issue detection rating. The methodbegins at step 590 where the analysis system selects and performs atleast two of steps 591-596. At step 591, the analysis system generates apolicy rating for the system aspect based on the issue detection dataand policy analysis parameters and in accordance with the at least oneevaluation perspective, the at least one evaluation viewpoint, andpolicy as the evaluation rating metric. At step 592, the analysis systemgenerates a documentation rating for the system aspect based on theissue detection data and documentation analysis parameters and inaccordance with the at least one evaluation perspective, the at leastone evaluation viewpoint, and documentation as the evaluation ratingmetric.

At step 593, the analysis system generates an automation rating for thesystem aspect based on the issue detection data and automation analysisparameters and in accordance with the at least one evaluationperspective, the at least one evaluation viewpoint, and automation asthe evaluation rating metric. At step 594, the analysis system generatesa policy rating for the system aspect based on the issue detection dataand policy analysis parameters and in accordance with the at least oneevaluation perspective, the at least one evaluation viewpoint, andpolicy as the evaluation rating metric.

At step 595, the analysis system generates a certification rating forthe system aspect based on the issue detection data and certificationanalysis parameters and in accordance with the at least one evaluationperspective, the at least one evaluation viewpoint, and certification asthe evaluation rating metric. At step 596, the analysis system generatesa procedure rating for the system aspect based on the issue detectiondata and procedure analysis parameters and in accordance with the atleast one evaluation perspective, the at least one evaluation viewpoint,and procedure as the evaluation rating metric.

The method continues at step 597 where the analysis system generates theissue detection rating based on the selected and performed at least twoof the process rating, the policy rating, the documentation rating, theautomation rating, the procedure rating, and the certification rating.For example, the issue detection rating is a summation of the at leastindividual evaluation metric ratings. As another example, the analysissystem performs a mathematical and/or logical function (e.g., a weightaverage, standard deviation, statistical analysis, trending, etc.) onthe at least two individual evaluation metric to produce the issuedetection rating.

FIG. 81 is a schematic block diagram of an embodiment of a scoringmodule of the data analysis module 252 that includes a process ratingmodule 601, a policy rating module 602, a procedure rating module 603, acertification rating module 604, a documentation rating module 605, anautomation rating module 606, and a cumulative rating module 607. Ingeneral, the data scoring module generates an issue detection rating 608from a collection of data based on data analysis parameters 265.

The process rating module 601 evaluates the collection of data 600, orportion thereof, (e.g., pre-processed data of FIG. 35) to produce aprocess evaluation rating in accordance with process analysis parametersof the data analysis parameters 265. The process analysis parametersindicate how the collection of data is to be evaluated with respect toprocesses of the system, or portion thereof. As an example, the processanalysis parameters include:

-   -   an instruction to compare processes of the data 600 with a list        of processes the system, or portion thereof, should have;    -   an instruction to count the number of processes of data 600 and        compare it with a quantity of processes the system, or portion        thereof, should have;    -   an instruction to determine last revisions of processes of data        600 and/or to determine an age of last revisions;    -   an instruction to determine frequency of use of processes of        data 600;    -   an instruction to determine a volume of access of processes of        data 600;    -   an instruction to evaluate a process of data 600 with respect to        a checklist regarding content of the process (e.g., what should        be in the process);    -   a scaling factor based on the size of the system, or portion        thereof;    -   a scaling factor based on the size of the organization;    -   an instruction to compare a balance of local processes with        respect to system-wide processes;    -   an instruction to compare topics of the processes of data 600        with desired topics for processes (which may be at least        partially derived from the evaluation category and/or        sub-categories); and/or    -   an instruction to evaluate language use within processes of data        600.

The process rating module 601 can rate the data 600 at three levels. Thefirst level is that the system has processes, the system has the rightnumber of processes, and/or the system has processes that address theright topics. The second level digs into the processes themselves todetermine whether they are adequately cover the requirements of thesystem. The third level evaluates how well the processes are used andhow well they are adhered to.

As an example, the process rating module 601 generates a processevaluation rating based on a comparison of the processes of the data 600with a list of processes the system, or portion thereof, should have. Ifall of the processes on the list are found in the data 600, then theprocess evaluation rating is high. The fewer processes on the list thatfound in the data 600, the lower the process evaluation rating will be.

As another example, the process rating module 601 generates a processevaluation rating based on a determination of the last revisions ofprocesses of data 600 and/or to determine an age of last revisions. As aspecific example, if processes are revised at a rate that corresponds toa rate of revision in the industry, then a relatively high processevaluation rate would be produced. As another specific example, ifprocesses are revised at a much lower rate that corresponds to a rate ofrevision in the industry, then a relatively low process evaluation ratewould be produced (implies a lack of attention to the processes). As yetanother specific example, if processes are revised at a much higher ratethat corresponds to a rate of revision in the industry, then arelatively low process evaluation rate would be produced (impliesprocesses are inaccurate, incomplete, and/or created with a lack ofknowledge as to what's needed).

As another example, the process rating module 601 generates a processevaluation rating based on a determination of frequency of use ofprocesses of data 600. As a specific example, if processes are used at afrequency (e.g., x times per week) that corresponds to a frequency ofuse in the industry, then a relatively high process evaluation ratewould be produced. As another specific example, if processes are used ata much lower frequency that corresponds to a frequency of use in theindustry, then a relatively low process evaluation rate would beproduced (implies a lack of using and adhering to the processes). As yetanother specific example, if processes are used at a much higherfrequency that corresponds to a frequency of use in the industry, then arelatively low process evaluation rate would be produced (impliesprocesses are inaccuracy, incompleteness, and/or difficult to use).

As another example, the process rating module 601 generates a processevaluation rating based on an evaluation of a process of data 600 withrespect to a checklist regarding content of the policy (e.g., whatshould be in the policy, which may be based, at least in part, on anevaluation category, sub-category, and/or sub-sub category). As aspecific example, the topics contained in the process of data 600 iscompared to a checklist of desired topics for such a process. If all ofthe topics on the checklist are found in the process of data 600, thenthe process evaluation rating is high. The fewer topics on the checklistthat found in the process of data 600, the lower the process evaluationrating will be.

As another example, the process rating module 601 generates a processevaluation rating based on a comparison of balance between localprocesses of data 600 and system-wide processes of data 600. As aspecific example, most security processes should be system-wide. Thus,if there are a certain percentage (e.g., less than 10%) of securityprocesses that are local, then a relatively high process evaluationrating will be generated. Conversely, the greater the percentage oflocal security processes, the lower the process evaluation rating willbe.

As another example, the process rating module 601 generates a processevaluation rating based on evaluation of language use within processesof data 600. As a specific example, most security requirements aremandatory. Thus, if the policy includes too much use of the word “may”(which implies optionality) versus the word “shall (which implies must),the lower the process evaluation rating will be.

The process rating module 601 may perform a plurality of the aboveexamples of process evaluation to produce a plurality of processevaluation ratings. The process rating module 601 may output theplurality of the process evaluation ratings to the cumulative ratingmodule 607. Alternatively, the process rating module 601 may perform afunction (e.g., a weight average, standard deviation, statisticalanalysis, etc.) on the plurality of process evaluation ratings toproduce a process evaluation rating that's provided to the cumulativerating module 607.

The policy rating module 602 evaluates the collection of data 600, orportion thereof, (e.g., pre-processed data of FIG. 35) to produce apolicy evaluation rating in accordance with policy analysis parametersof the data analysis parameters 265. The policy analysis parametersindicate how the collection of data is to be evaluated with respect topolicies of the system, or portion thereof. As an example, the policyanalysis parameters include:

-   -   an instruction to compare policies of the data 600 with a list        of policies the system, or portion thereof, should have;    -   an instruction to count the number of policies of data 600 and        compare it with a quantity of policies the system, or portion        thereof, should have;    -   an instruction to determine last revisions of policies of data        600 and/or to determine an age of last revisions;    -   an instruction to determine frequency of use of policies of data        600;    -   an instruction to determine a volume of access of policies of        data 600;    -   an instruction to evaluate a policy of data 600 with respect to        a checklist regarding content of the policy (e.g., what should        be in the policy);    -   a scaling factor based on the size of the system, or portion        thereof;    -   a scaling factor based on the size of the organization;    -   an instruction to compare a balance of local policies with        respect to system-wide policies;    -   an instruction to compare topics of the policies of data 600        with desired topics for policies (which may be at least        partially derived from the evaluation category and/or        sub-categories); and/or    -   an instruction to evaluate language use within policies of data        600.

The policy rating module 602 can rate the data 600 at three levels. Thefirst level is that the system has policies, the system has the rightnumber of policies, and/or the system has policies that address theright topics. The second level digs into the policies themselves todetermine whether they are adequately cover the requirements of thesystem. The third level evaluates how well the policies are used and howwell they are adhered to.

The procedure rating module 603 evaluates the collection of data 600, orportion thereof, (e.g., pre-processed data of FIG. 35) to produce aprocedure evaluation rating in accordance with procedure analysisparameters of the data analysis parameters 265. The procedure analysisparameters indicate how the collection of data is to be evaluated withrespect to procedures of the system, or portion thereof. As an example,the procedure analysis parameters include:

-   -   an instruction to compare procedures of the data 600 with a list        of procedures the system, or portion thereof, should have;    -   an instruction to count the number of procedures of data 600 and        compare it with a quantity of procedures the system, or portion        thereof, should have;    -   an instruction to determine last revisions of procedures of data        600 and/or to determine an age of last revisions;    -   an instruction to determine frequency of use of procedures of        data 600;    -   an instruction to determine a volume of access of procedures of        data 600;    -   an instruction to evaluate a procedure of data 600 with respect        to a checklist regarding content of the procedure (e.g., what        should be in the procedure);    -   a scaling factor based on the size of the system, or portion        thereof;    -   a scaling factor based on the size of the organization;    -   an instruction to compare a balance of local procedures with        respect to system-wide procedures;    -   an instruction to compare topics of the procedures of data 600        with desired topics for procedures (which may be at least        partially derived from the evaluation category and/or        sub-categories); and/or    -   an instruction to evaluate language use within procedures of        data 600.

The procedure rating module 603 can rate the data 600 at three levels.The first level is that the system has procedures, the system has theright number of procedures, and/or the system has procedures thataddress the right topics. The second level digs into the proceduresthemselves to determine whether they are adequately cover therequirements of the system. The third level evaluates how well theprocedures are used and how well they are adhered to.

The certification rating module 604 evaluates the collection of data600, or portion thereof, (e.g., pre-processed data of FIG. 35) toproduce a certification evaluation rating in accordance withcertification analysis parameters of the data analysis parameters 265.The certification analysis parameters indicate how the collection ofdata is to be evaluated with respect to certifications of the system, orportion thereof. As an example, the certification analysis parametersinclude:

-   -   an instruction to compare certifications of the data 600 with a        list of certifications the system, or portion thereof, should        have;    -   an instruction to count the number of certifications of data 600        and compare it with a quantity of certifications the system, or        portion thereof, should have;    -   an instruction to determine last revisions of certifications of        data 600 and/or to determine an age of last revisions;    -   an instruction to evaluate a certification of data 600 with        respect to a checklist regarding content of the certification        (e.g., what should be certified and/or how it should be        certified);    -   a scaling factor based on the size of the system, or portion        thereof;    -   a scaling factor based on the size of the organization; and    -   an instruction to compare a balance of local certifications with        respect to system-wide certifications.

The certification rating module 603 can rate the data 600 at threelevels. The first level is that the system has certifications, thesystem has the right number of certifications, and/or the system hascertifications that address the right topics. The second level digs intothe certifications themselves to determine whether they are adequatelycover the requirements of the system. The third level evaluates how wellthe certifications are maintained and updated.

The documentation rating module 603 evaluates the collection of data600, or portion thereof, (e.g., pre-processed data of FIG. 35) toproduce a documentation evaluation rating in accordance withdocumentation analysis parameters of the data analysis parameters 265.The documentation analysis parameters indicate how the collection ofdata is to be evaluated with respect to documentation of the system, orportion thereof. As an example, the documentation analysis parametersinclude:

-   -   an instruction to compare documentation of the data 600 with a        list of documentation the system, or portion thereof, should        have;    -   an instruction to count the number of documentation of data 600        and compare it with a quantity of documentation the system, or        portion thereof, should have;    -   an instruction to determine last revisions of documentation of        data 600 and/or to determine an age of last revisions;    -   an instruction to determine frequency of use and/or creation of        documentation of data 600;    -   an instruction to determine a volume of access of documentation        of data 600;    -   an instruction to evaluate a document of data 600 with respect        to a checklist regarding content of the document (e.g., what        should be in the document);    -   a scaling factor based on the size of the system, or portion        thereof;    -   a scaling factor based on the size of the organization;    -   an instruction to compare a balance of local documents with        respect to system-wide documents;    -   an instruction to compare topics of the documentation of data        600 with desired topics for documentation (which may be at least        partially derived from the evaluation category and/or        sub-categories); and/or    -   an instruction to evaluate language use within documentation of        data 600.

The documentation rating module 605 can rate the data 600 at threelevels. The first level is that the system has documentation, the systemhas the right number of documents, and/or the system has documents thataddress the right topics. The second level digs into the documentsthemselves to determine whether they are adequately cover therequirements of the system. The third level evaluates how well thedocumentation is used and how well it is maintained.

The automation rating module 606 evaluates the collection of data 600,or portion thereof, (e.g., pre-processed data of FIG. 35) to produce anautomation evaluation rating in accordance with automation analysisparameters of the data analysis parameters 265. The automation analysisparameters indicate how the collection of data is to be evaluated withrespect to automation of the system, or portion thereof. As an example,the automation analysis parameters include:

-   -   an instruction to compare automation of the data 600 with a list        of automation the system, or portion thereof, should have;    -   an instruction to count the number of automation of data 600 and        compare it with a quantity of automation the system, or portion        thereof, should have;    -   an instruction to determine last revisions of automation of data        600 and/or to determine an age of last revisions;    -   an instruction to determine frequency of use of automation of        data 600;    -   an instruction to determine a volume of access of automation of        data 600;    -   an instruction to evaluate an automation of data 600 with        respect to a checklist regarding content of the automation        (e.g., what the automation should do);    -   a scaling factor based on the size of the system, or portion        thereof;    -   a scaling factor based on the size of the organization;    -   an instruction to compare a balance of local automation with        respect to system-wide automation;    -   an instruction to compare topics of the automation of data 600        with desired topics for automation (which may be at least        partially derived from the evaluation category and/or        sub-categories); and/or    -   an instruction to evaluate operation use of automation of data        600.

The automation rating module 606 can rate the data 600 at three levels.The first level is that the system has automation, the system has theright number of automation, and/or the system has automation thataddress the right topics. The second level digs into the automationthemselves to determine whether they are adequately cover therequirements of the system. The third level evaluates how well theautomations are used and how well they are adhered to.

The cumulative rating module 607 receives one or more process evaluationratings, one or more policy evaluation ratings, one or more procedureevaluation ratings, one or more certification evaluation ratings, one ormore documentation evaluation ratings, and/or one or more automationevaluation ratings. The cumulative rating module 607 may output theevaluation ratings it receives as the issue detection rating 608.Alternatively, the cumulative rating module 607 performs a function(e.g., a weight average, standard deviation, statistical analysis, etc.)on the evaluation ratings it receives to produce the issue detectionrating 608.

FIG. 82 is a schematic block diagram of another embodiment of a dataanalysis module 252 that is similar to the data analysis module of FIG.81. In this embodiment, the data analysis module 252 includes a dataparsing module 609, which parses the data 600 into process data, policydata, procedure data, certification data, documentation data, and/orautomation data.

FIG. 83 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular generating a process rating. The method begins at step 610where the analysis system generates a first process rating based on afirst combination of a system criteria (e.g., system requirements), of asystem mode (e.g., system functions), of an evaluation perspective(e.g., implementation), and of an evaluation viewpoint (e.g., discloseddata).

The method continues at step 611 where the analysis system generates asecond process rating based on a second combination of a system criteria(e.g., system design), of a system mode (e.g., system functions), of anevaluation perspective (e.g., implementation), and of an evaluationviewpoint (e.g., disclosed data). The method continues at step 612 wherethe analysis system generates the process rating based on the first andsecond process ratings.

FIG. 84 is a logic diagram of a further example of generating a processrating for understanding of system build for issue detection securityfunctions of an organization. The method begins at step 613 where theanalysis system identifies processes regarding issue detection securityfunctions from the data. The method continues at step 614 where theanalysis system generates a process rating from the data. Examples ofthis were discussed with reference to FIG. 81.

The method also continues at step 615 where the analysis systemdetermines use of the issue detection processes. The method continues atstep 616 where the analysis system generates a process rating based onuse. Examples of this were discussed with reference to FIG. 81.

The method also continues at step 617 where the analysis systemdetermines consistency of applying the issue detection processes. Themethod continues at step 618 where the analysis system generates aprocess rating based on consistency of use. Examples of this werediscussed with reference to FIG. 81. The method continues at step 619where the analysis system generates the process rating based on theprocess rating from the data, the process rating based on use, and theprocess rating based on consistency of use.

FIG. 85 is a logic diagram of a further example of generating a processrating for understanding of verifying issue detection security functionsof an organization. The method begins at step 620 where the analysissystem identifies processes to verify issue detection security functionsfrom the data. The method continues at step 621 where the analysissystem generates a process rating from the data. Examples of this werediscussed with reference to FIG. 81.

The method also continues at step 622 where the analysis systemdetermines use of the processes to verify the issue detection securityfunctions. The method continues at step 623 where the analysis systemgenerates a process rating based on use of the verify processes.Examples of this were discussed with reference to FIG. 81.

The method also continues at step 624 where the analysis systemdetermines consistency of applying the verifying processes to issuedetection security functions. The method continues at step 625 where theanalysis system generates a process rating based on consistency of use.Examples of this were discussed with reference to FIG. 81. The methodcontinues at step 626 where the analysis system generates the processrating based on the process rating from the data, the process ratingbased on use, and the process rating based on consistency of use.

FIG. 86 is a diagram of an example of issue detection data for use by ananalysis system to generate an issue detection rating for a system, orportion thereof. The issue detection data includes a collection of data600, which includes disclosed data, discovered data, and/or desireddata.

For example, data 600 includes one or more network operation and dataflow establishment and management processes, one or more data protectiontools, one or more vulnerability management tools, one or more networkmonitoring tools, one or more access management tools, one or moreboundary protection tools, network operation and data flow establishmentand management implementation, device information, one or more systemplans, one or more diagrams, one or more event detection analysisprocesses, one or more event detection analysis tools, event detectionanalysis implementation, one or more event data aggregation andcorrelation processes, one or more event data aggregation andcorrelation tools, event data aggregation and correlationimplementation, one or more logs, one or more event impact determinationprocesses, one or more event impact determination tools, event impactdetermination implementation, one or more incident alert thresholdestablishment processes, one or more incident alert thresholdestablishment tools, incident alert threshold establishmentimplementation, one or more network monitoring processes, networkmonitoring implementation, one or more network physical environmentmonitoring processes, one or more physical environment monitoring tools,physical environment monitoring implementation, one or more personnelactivity monitoring processes, one or more personnel activity monitoringtools, personnel activity monitoring implementation, one or moremalicious code detection processes, one or more malware protectiontools, one or more exploit protection tools, malicious code detectionimplementation, one or more unauthorized mobile code detectionprocesses, one or more unauthorized mobile code detection tools,unauthorized mobile code detection implementation, one or more externalservice provider monitoring processes, one or more external serviceprovider monitoring tools, external service provider monitoringimplementation, one or more unauthorized aspect (e.g., personnel,connections, devices, and software) monitoring processes, unauthorizedaspect monitoring implementation, one or more vulnerability scanningprocesses, vulnerability scanning implementation, one or moreverification processes, one or more verification tools, assigned issuedetection roles and responsibilities, user information, one or moretraining documents, one or more issue detection process complianceverification processes, one or more reports, one or more issue detectionprocess testing processes, one or more issue detection processcommunication processes, and/or one or more issue detection processimprovement processes.

The data 600 may further include one or more network operation and dataflow establishment and management policies, one or more event detectionanalysis policies, one or more event data aggregation and correlationpolicies, one or more event impact determination policies, one or moreincident alert threshold establishment policies, one or more networkmonitoring policies, one or more network physical environment monitoringpolicies, one or more personnel activity monitoring policies, one ormore malicious code detection policies, one or more unauthorized mobilecode detection policies, one or more external service providermonitoring policies, one or more unauthorized aspect (e.g., personnel,connections, devices, and software) monitoring policies, one or morevulnerability scanning policies, one or more verification policies, oneor more issue detection process compliance verification policies, one ormore issue detection process testing policies, one or more issuedetection process communication policies, and/or one or more issuedetection process improvement policies.

The data 600 may still include one or more network operation and dataflow establishment and management procedures, one or more eventdetection analysis procedures, one or more event data aggregation andcorrelation procedures, one or more event impact determinationprocedures, one or more incident alert threshold establishmentprocedures, one or more network monitoring procedures, one or morenetwork physical environment monitoring procedures, one or morepersonnel activity monitoring procedures, one or more malicious codedetection procedures, one or more unauthorized mobile code detectionprocedures, one or more external service provider monitoring procedures,one or more unauthorized aspect (e.g., personnel, connections, devices,and software) monitoring procedures, one or more vulnerability scanningprocedures, one or more verification procedures, one or more issuedetection process compliance verification procedures, one or more issuedetection process testing procedures, one or more issue detectionprocess communication procedures, and/or one or more issue detectionprocess improvement procedures.

The data 600 may still include one or more network operation and dataflow establishment and management documents, one or more event detectionanalysis documents, one or more event data aggregation and correlationdocuments, one or more event impact determination documents, one or moreincident alert threshold establishment documents, one or more networkmonitoring documents, one or more network physical environmentmonitoring documents, one or more personnel activity monitoringdocuments, one or more malicious code detection documents, one or moreunauthorized mobile code detection documents, one or more externalservice provider monitoring documents, one or more unauthorized aspect(e.g., personnel, connections, devices, and software) monitoringdocuments, one or more vulnerability scanning documents, one or moreverification documents, one or more issue detection process complianceverification documents, one or more issue detection process testingdocuments, one or more issue detection process communication documents,and/or one or more issue detection process improvement documents.

The data 600 may still include one or more network operation and dataflow establishment and management certifications, one or more eventdetection analysis certifications, one or more event data aggregationand correlation certifications, one or more event impact determinationcertifications, one or more incident alert threshold establishmentcertifications, one or more network monitoring certifications, one ormore network physical environment monitoring certifications, one or morepersonnel activity monitoring certifications, one or more malicious codedetection certifications, one or more unauthorized mobile code detectioncertifications, one or more external service provider monitoringcertifications, one or more unauthorized aspect (e.g., personnel,connections, devices, and software) monitoring certifications, one ormore vulnerability scanning certifications, one or more verificationcertifications, one or more issue detection process complianceverification certifications, one or more issue detection process testingcertifications, one or more issue detection process communicationcertifications, and/or one or more issue detection process improvementcertifications.

The data 600 may still include one or more network operation and dataflow establishment and management automations, one or more eventdetection analysis automations, one or more event data aggregation andcorrelation automations, one or more event impact determinationautomations, one or more incident alert threshold establishmentautomations, one or more network monitoring automations, one or morenetwork physical environment monitoring automations, one or morepersonnel activity monitoring automations, one or more malicious codedetection automations, one or more unauthorized mobile code detectionautomations, one or more external service provider monitoringautomations, one or more unauthorized aspect (e.g., personnel,connections, devices, and software) monitoring automations, one or morevulnerability scanning automations, one or more verificationautomations, one or more issue detection process compliance verificationautomations, one or more issue detection process testing automations,one or more issue detection process communication automations, and/orone or more issue detection process improvement automations.

In this example the blue shaded boxes (e.g., network operation and dataflow establishment and management processes, event detection analysisprocesses, etc.) are data that is directly relevant to the processrating module 601. The light green shaded boxes (e.g., data protectiontools, vulnerability management tools, etc.) are data that may berelevant to the process rating module 601. Implementations (e.g.,network operation and data flow establishment and managementimplementation, event detection analysis implementation, etc.) refer tohow a particular process is performed within the system aspect. Otherdata (e.g., diagrams, reports, etc.) provides information regarding oneor more of the processes, tools, and implementations. The data listed isexemplary and not intended to be an exhaustive list.

There may be overlap and/or redundancies with respect to the processes,tools, and implementations listed. For example, a security informationevent management (SIEM) tool may include a combination of tools such asvulnerability management, vulnerability scanning, network monitoring,event detection analysis, event aggregation and correlation, eventimpact determination, etc. As another example, a network monitoring toolmay be considered a boundary protection tool and vice versa. Tools maybe shared across processes. For example, while vulnerability managementtools are listed near network operation and data flow establishment andmanagement processes, several other processes may rely on vulnerabilitymanagement tools such as vulnerability scanning processes and eventawareness analysis processes.

In one embodiment, the process rating module 601 rates how wellprocesses are used for identified tasks. For example, the process ratingmodule 601 rates how well the network operation and data flowestablishment and management tools (e.g., data protection tools,vulnerability management tools, network monitoring tools, accessmanagement tools, boundary protection tools, etc.) are used inaccordance with the network operation and data flow establishment andmanagement processes.

In another embodiment, the process rating module 601 rates theconsistency of application of processes in implementation. For example,the process rating module rates the consistency of use of the networkoperation and data flow establishment and management processes to usethe network operation and data flow establishment and management toolsin the network operation and data flow establishment and managementimplementation (e.g., to establish and manage the network operation andexpected data flows of the system).

FIG. 87 is a logic diagram of an example of generating a process ratingby the analysis system; in particular the process rating modulegenerating a process rating based on data. The method begins at step 620where the analysis system determines whether there is at least oneprocess in the collection of data. Note that the threshold number inthis step could be greater than one. If there are no processes, themethod continues at step 631 where the analysis system generates aprocess rating of 0 (and/or a word rating of “none”).

If there is at least one process, the method continues at step 632 wherethe analysis system determines whether the processes are repeatable. Inthis instance, repeatable processes produce consistent results, includevariations from process to process, are not routinely reviewed in anorganized manner, and/or are not all regulated. For example, when thenumber of processes is below a desired number of processes, the analysissystem determines that the processes are not repeatable (e.g., with toofew processes cannot get repeatable outcomes). As another example, whenthe processes of the data 600 does not include one or more processes ona list of processes the system should have, the analysis systemdetermines that the processes are not repeatable (e.g., with missingprocesses cannot get repeatable outcomes).

If the processes are not repeatable, the method continues at step 633where the analysis system generates a process rating of 10 (and/or aword rating of “inconsistent”). If, however, the processes are at leastrepeatable, the method continues at step 634 where the analysis systemdetermines whether the processes are standardized. In this instance,standardized includes repeatable plus there are no appreciablevariations in the processes from process to process, and/or theprocesses are regulated.

If the processes are not standardized, the method continues at step 635where the analysis system generates a process rating of 20 (and/or aword rating of “repeatable”). If, however, the processes are at leaststandardized, the method continues at step 636 where the analysis systemdetermines whether the processes are measured. In this instance,measured includes standardized plus precise, exact, and/or calculated tospecific needs, concerns, and/or functioning of the system.

If the processes are not measured, the method continues at step 637where the analysis system generates a process rating of 30 (and/or aword rating of “standardized”). If, however, the processes are at leastmeasured, the method continues at step 638 where the analysis systemdetermines whether the processes are optimized. In this instance,optimized includes measured plus processes are up-to-date and/or processimprovement assessed on a regular basis as part of system protocols.

If the processes are not optimized, the method continues at step 639where the analysis system generates a process rating of 40 (and/or aword rating of “measured”). If the processes are optimized, the methodcontinues at step 640 where the analysis system generates a processrating of 50 (and/or a word rating of “optimized”). Note that thenumerical ratings are example values and could be other values. Furthernote that the number of level of process rating may be more or less thanthe six shown.

For this method, distinguishing between repeatable, standardized,measured, and optimized is interpretative based on the manner in whichthe data 600 was analyzed. As an example, weighting factors on certaintypes of analysis affect the level. As a specific example, weightingfactors for analysis to determine last revisions of processes, age oflast revisions, content verification of processes with respect to achecklist, balance of local processes and system-wide processes, topicverification of the processes with respect to desired topics, and/orprocess language evaluation will affect the resulting level.

FIG. 88 is a logic diagram of an example of generating a process ratingby the analysis system; in particular the process rating modulegenerating a process rating based on use of processes. The method beginsat step 641 where the analysis system determines whether at least oneprocess in the collection of data has been used. Note that the thresholdnumber in this step could be greater than one. If no processes have beenused, the method continues at step 642 where the analysis systemgenerates a process rating of 0 (and/or a word rating of “none”).

If at least one process is used, the method continues at step 643 wherethe analysis system determines whether the use of the processes isrepeatable. In this instance, repeatable use of processes is consistentuse, but with variations from process to process, use is not routinelyreviewed or verified in an organized manner, and/or use is notregulated.

If the use of processes is not repeatable, the method continues at step644 where the analysis system generates a process rating of 10 (and/or aword rating of “inconsistent”). If, however, the use of processes is atleast repeatable, the method continues at step 645 where the analysissystem determines whether the use of processes is standardized. In thisinstance, standardized includes repeatable plus there are no appreciablevariations in the use of processes from process to process, and/or theuse of processes is regulated.

If the use of processes is not standardized, the method continues atstep 646 where the analysis system generates a process rating of 20(and/or a word rating of “repeatable”). If, however, the use ofprocesses is at least standardized, the method continues at step 647where the analysis system determines whether the use of processes ismeasured. In this instance, measured includes standardized plus use isprecise, exact, and/or calculated to specific needs, concerns, and/orfunctioning of the system.

If the use of processes is not measured, the method continues at step648 where the analysis system generates a process rating of 30 (and/or aword rating of “standardized”). If, however, the use of processes is atleast measured, the method continues at step 649 where the analysissystem determines whether the use of processes is optimized. In thisinstance, optimized includes measured plus use of processes areup-to-date and/or improving use of processes is assessed on a regularbasis as part of system protocols.

If the use of processes is not optimized, the method continues at step650 where the analysis system generates a process rating of 40 (and/or aword rating of “measured”). If the use of processes is optimized, themethod continues at step 651 where the analysis system generates aprocess rating of 50 (and/or a word rating of “optimized”). Note thatthe numerical ratings are example values and could be other values.Further note that the number of level of process rating may be more orless than the six shown.

FIG. 89 is a logic diagram of an example of generating a process ratingby the analysis system; in particular the process rating modulegenerating a process rating based on consistency of application ofprocesses. The method begins at step 652 where the analysis systemdetermines whether at least one process in the collection of data hasbeen consistently applied. Note that the threshold number in this stepcould be greater than one. If there no processes have been consistentlyapplied, the method continues at step 653 where the analysis systemgenerates a process rating of 0 (and/or a word rating of “none”).

If at least one process has been consistently applied, the methodcontinues at step 654 where the analysis system determines whether theconsistent application of processes is repeatable. In this instance,repeatable consistency of application of processes is a process isconsistently applied for a given circumstance of the system (e.g.,determining software applications for like devices in a department), butwith variations from process to process, application of processes is notroutinely reviewed or verified in an organized manner, and/orapplication of processes is not regulated.

If the consistency of application of processes is not repeatable, themethod continues at step 655 where the analysis system generates aprocess rating of 10 (and/or a word rating of “inconsistent”). If,however, the consistency of application of processes is at leastrepeatable, the method continues at step 656 where the analysis systemdetermines whether the consistency of application of processes isstandardized. In this instance, standardized includes repeatable plusthere are no appreciable variations in the application of processes fromprocess to process, and/or the application of processes is regulated.

If the consistency of application of processes is not standardized, themethod continues at step 657 where the analysis system generates aprocess rating of 20 (and/or a word rating of “repeatable”). If,however, the consistency of application of processes is at leaststandardized, the method continues at step 658 where the analysis systemdetermines whether the consistency of application of processes ismeasured. In this instance, measured includes standardized plusapplication of processes is precise, exact, and/or calculated tospecific needs, concerns, and/or functioning of the system.

If the consistency of application of processes is not measured, themethod continues at step 659 where the analysis system generates aprocess rating of 30 (and/or a word rating of “standardized”). If,however, the consistency of application of processes is at leastmeasured, the method continues at step 660 where the analysis systemdetermines whether the consistency of application of processes isoptimized. In this instance, optimized includes measured plusapplication of processes is up-to-date and/or improving application ofprocesses is assessed on a regular basis as part of system protocols.

If the consistency of application of processes is not optimized, themethod continues at step 661 where the analysis system generates aprocess rating of 40 (and/or a word rating of “measured”). If theconsistency of application of processes is optimized, the methodcontinues at step 662 where the analysis system generates a processrating of 50 (and/or a word rating of “optimized”). Note that thenumerical ratings are example values and could be other values. Furthernote that the number of level of process rating may be more or less thanthe six shown.

FIG. 90 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular generating a policy rating. The method begins at step 670where the analysis system generates a first policy rating based on afirst combination of a system criteria (e.g., system requirements), of asystem mode (e.g., system functions), of an evaluation perspective(e.g., implementation), and of an evaluation viewpoint (e.g., discloseddata).

The method continues at step 671 where the analysis system generates asecond policy rating based on a second combination of a system criteria(e.g., system design), of a system mode (e.g., system functions), of anevaluation perspective (e.g., implementation), and of an evaluationviewpoint (e.g., disclosed data). The method continues at step 672 wherethe analysis system generates the policy rating based on the first andsecond policy ratings.

FIG. 91 is a logic diagram of a further example of generating a policyrating for understanding of system build for issue detection securityfunctions of an organization. The method begins at step 673 where theanalysis system identifies policies regarding issue detection securityfunctions from the data. The method continues at step 674 where theanalysis system generates a policy rating from the data. Examples ofthis were discussed with reference to FIG. 81.

The method also continues at step 675 where the analysis systemdetermines use of the policies for issue detection. The method continuesat step 676 where the analysis system generates a policy rating based onuse. Examples of this were discussed with reference to FIG. 81.

The method also continues at step 677 where the analysis systemdetermines consistency of applying the policies for issue detection. Themethod continues at step 678 where the analysis system generates apolicy rating based on consistency of use. Examples of this werediscussed with reference to FIG. 81. The method continues at step 679where the analysis system generates the policy rating based on thepolicy rating from the data, the policy rating based on use, and thepolicy rating based on consistency of use.

FIG. 92 is a logic diagram of a further example of generating a policyrating for understanding of verifying issue detection security functionsof an organization. The method begins at step 680 where the analysissystem identifies policies to verify the issue detection securityfunctions from the data. The method continues at step 681 where theanalysis system generates a policy rating from the data. Examples ofthis were discussed with reference to FIG. 81.

The method also continues at step 682 where the analysis systemdetermines use of the policies to verify the issue detection securityfunctions. The method continues at step 683 where the analysis systemgenerates a policy rating based on use of the verify policies. Examplesof this were discussed with reference to FIG. 81.

The method also continues at step 684 where the analysis systemdetermines consistency of applying the verifying policies to issuedetection security functions. The method continues at step 685 where theanalysis system generates a policy rating based on consistency of use.Examples of this were discussed with reference to FIG. 81. The methodcontinues at step 686 where the analysis system generates the policyrating based on the policy rating from the data, the policy rating basedon use, and the policy rating based on consistency of use.

FIG. 93 is a logic diagram of an example of generating a policy ratingby the analysis system; in particular the policy rating modulegenerating a policy rating based on data 600. The method begins at step687 where the analysis system determines whether there is at least onepolicy in the collection of data. Note that the threshold number in thisstep could be greater than one. If there are no policies, the methodcontinues at step 688 where the analysis system generates a policyrating of 0 (and/or a word rating of “none”).

If there is at least one policy, the method continues at step 689 wherethe analysis system determines whether the policies are defined. In thisinstance, defined policies include sufficient detail to produceconsistent results, include variations from policy to policy, are notroutinely reviewed in an organized manner, and/or are not all regulated.For example, when the number of policies is below a desired number ofpolicies, the analysis system determines that the processes are notrepeatable (e.g., with too few policies cannot get repeatable outcomes).As another example, when the policies of the data 600 does not includeone or more policies on a list of policies the system should have, theanalysis system determines that the policies are not repeatable (e.g.,with missing policies cannot get repeatable outcomes).

If the policies are not defined, the method continues at step 690 wherethe analysis system generates a policy rating of 5 (and/or a word ratingof “informal”). If, however, the policies are at least defined, themethod continues at step 691 where the analysis system determineswhether the policies are audited. In this instance, audited includesdefined plus the policies are routinely reviewed, and/or the policiesare regulated.

If the policies are not audited, the method continues at step 692 wherethe analysis system generates a policy rating of 10 (and/or a wordrating of “defined”). If, however, the policies are at least audited,the method continues at step 693 where the analysis system determineswhether the policies are embedded. In this instance, embedded includesaudited plus are systematically rooted in most, if not all, aspects ofthe system.

If the policies are not embedded, the method continues at step 694 wherethe analysis system generates a policy rating of 15 (and/or a wordrating of “audited”). If the policies are embedded, the method continuesat step 695 where the analysis system generates a policy rating of 20(and/or a word rating of “embedded”). Note that the numerical ratingsare example values and could be other values. Further note that thenumber of level of policy rating may be more or less than the fiveshown.

For this method, distinguishing between defined, audited, and embeddedis interpretative based on the manner in which the data 600 wasanalyzed. As an example, weighting factors on certain types of analysisaffect the level. As a specific example, weighting factors for analysisto determine last revisions of policies, age of last revisions, contentverification of policies with respect to a checklist, balance of localpolicies and system-wide policies, topic verification of the policieswith respect to desired topics, and/or policy language evaluation willaffect the resulting level.

FIG. 94 is a logic diagram of an example of generating a policy ratingby the analysis system; in particular the policy rating modulegenerating a policy rating based on use of the polices. The methodbegins at step 696 where the analysis system determines whether there isat least one use of a policy. Note that the threshold number in thisstep could be greater than one. If there are no uses of policies, themethod continues at step 697 where the analysis system generates apolicy rating of 0 (and/or a word rating of “none”).

If there is at least one use of a policy, the method continues at step698 where the analysis system determines whether the use of policies isdefined. In this instance, defined use of policies include sufficientdetail on how and/or when to use a policy, include variations in usefrom policy to policy, use of policies is not routinely reviewed in anorganized manner, and/or use of policies is not regulated.

If the use of policies is not defined, the method continues at step 699where the analysis system generates a policy rating of 5 (and/or a wordrating of “informal”). If, however, the use of policies is at leastdefined, the method continues at step 700 where the analysis systemdetermines whether the use of policies is audited. In this instance,audited includes defined plus the use of policies is routinely reviewed,and/or the use of policies is regulated.

If the use of policies is not audited, the method continues at step 701where the analysis system generates a policy rating of 10 (and/or a wordrating of “defined”). If, however, the use of policies is at leastaudited, the method continues at step 702 where the analysis systemdetermines whether the use of policies is embedded. In this instance,embedded includes audited plus use of policies is systematically rootedin most, if not all, aspects of the system.

If the use of policies is not embedded, the method continues at step 703where the analysis system generates a policy rating of 15 (and/or a wordrating of “audited”). If the use of policies is embedded, the methodcontinues at step 704 where the analysis system generates a policyrating of 20 (and/or a word rating of “embedded”). Note that thenumerical ratings are example values and could be other values. Furthernote that the number of level of policy rating may be more or less thanthe five shown.

FIG. 95 is a logic diagram of an example of generating a policy ratingby the analysis system; in particular the policy rating modulegenerating a policy rating based on consistent application of polices.The method begins at step 705 where the analysis system determineswhether there is at least one consistent application of a policy. Notethat the threshold number in this step could be greater than one. Ifthere are no consistent application of policies, the method continues atstep 706 where the analysis system generates a policy rating of 0(and/or a word rating of “none”).

If there is at least one consistent application of a policy, the methodcontinues at step 707 where the analysis system determines whether theconsistent application of policies is defined. In this instance, definedapplication of policies include sufficient detail on when policiesapply, includes application variations from policy to policy,application of policies is not routinely reviewed in an organizedmanner, and/or application of policies is not regulated.

If the application of policies is not defined, the method continues atstep 708 where the analysis system generates a policy rating of 5(and/or a word rating of “informal”). If, however, the application ofpolicies is at least defined, the method continues at step 707 where theanalysis system determines whether the application of policies isaudited. In this instance, audited includes defined plus the applicationof policies is routinely reviewed, and/or the application of policies isregulated.

If the application of policies is not audited, the method continues atstep 710 where the analysis system generates a policy rating of 10(and/or a word rating of “defined”). If, however, the application ofpolicies is at least audited, the method continues at step 711 where theanalysis system determines whether the application of policies isembedded. In this instance, embedded includes audited plus applicationof policies is systematically rooted in most, if not all, aspects of thesystem.

If the application of policies is not embedded, the method continues atstep 712 where the analysis system generates a policy rating of 15(and/or a word rating of “audited”). If the application of policies isembedded, the method continues at step 713 where the analysis systemgenerates a policy rating of 20 (and/or a word rating of “embedded”).Note that the numerical ratings are example values and could be othervalues. Further note that the number of level of policies may be more orless than the five shown.

FIG. 96 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular generating a documentation rating. The method begins atstep 720 where the analysis system generates a first documentationrating based on a first combination of a system criteria (e.g., systemrequirements), of a system mode (e.g., system functions), of anevaluation perspective (e.g., implementation), and of an evaluationviewpoint (e.g., disclosed data).

The method continues at step 721 where the analysis system generates asecond documentation rating based on a second combination of a systemcriteria (e.g., system design), of a system mode (e.g., systemfunctions), of an evaluation perspective (e.g., implementation), and ofan evaluation viewpoint (e.g., disclosed data). The method continues atstep 722 where the analysis system generates the documentation ratingbased on the first and second documentation ratings.

FIG. 97 is a logic diagram of a further example of generating adocumentation rating for understanding of system build of issuedetection security functions of an organization. The method begins atstep 723 where the analysis system identifies documentation regardingissue detection security functions from the data. The method continuesat step 724 where the analysis system generates a documentation ratingfrom the data. Examples of this were discussed with reference to FIG.81.

The method also continues at step 725 where the analysis systemdetermines use of the documentation for issue detection securityfunctions. The method continues at step 726 where the analysis systemgenerates a documentation rating based on use. Examples of this werediscussed with reference to FIG. 81.

The method also continues at step 727 where the analysis systemdetermines consistency of applying the documentation for issue detectionsecurity functions. The method continues at step 728 where the analysissystem generates a documentation rating based on consistency of use.Examples of this were discussed with reference to FIG. 81. The methodcontinues at step 729 where the analysis system generates thedocumentation rating based on the documentation rating from the data,the documentation rating based on use, and the documentation ratingbased on consistency of use.

FIG. 98 is a logic diagram of a further example of generating adocumentation rating for understanding of system build of issuedetection security functions of an organization. The method begins atstep 730 where the analysis system identifies documentation to verifythe issue detection security functions from the data. The methodcontinues at step 731 where the analysis system generates adocumentation rating from the data. Examples of this were discussed withreference to FIG. 81.

The method also continues at step 732 where the analysis systemdetermines use of the documentation to verify issue detection securityfunctions. The method continues at step 733 where the analysis systemgenerates a documentation rating based on use of the verifydocumentation. Examples of this were discussed with reference to FIG.81.

The method also continues at step 734 where the analysis systemdetermines consistency of applying the verifying documentation for issuedetection security functions. The method continues at step 735 where theanalysis system generates a documentation rating based on consistency ofuse. Examples of this were discussed with reference to FIG. 81. Themethod continues at step 736 where the analysis system generates thedocumentation rating based on the documentation rating from the data,the documentation rating based on use, and the documentation ratingbased on consistency of use.

FIG. 99 is a logic diagram of an example of generating a documentationrating by the analysis system; in particular the documentation ratingmodule generating a documentation rating based on data 600. The methodbegins at step 737 where the analysis system determines whether there isat least one document in the collection of data. Note that the thresholdnumber in this step could be greater than one. If there are nodocuments, the method continues at step 738 where the analysis systemgenerates a documentation rating of 0 (and/or a word rating of “none”).

If there is at least one document, the method continues at step 739where the analysis system determines whether the documents areformalized. In this instance, formalized documents include sufficientdetail to produce consistent documentation, include form variations fromdocument to document, are not routinely reviewed in an organized manner,and/or formation of documents is not regulated.

If the documents are not formalized, the method continues at step 740where the analysis system generates a documentation rating of 5 (and/ora word rating of “informal”). If, however, the documents are at leastformalized, the method continues at step 741 where the analysis systemdetermines whether the documents are metric & reporting. In thisinstance, metric & reporting includes formal plus the documents areroutinely reviewed, and/or the formation of documents is regulated.

If the documents are not metric & reporting, the method continues atstep 742 where the analysis system generates a documentation rating of10 (and/or a word rating of “formal”). If, however, the documents are atleast metric & reporting, the method continues at step 743 where theanalysis system determines whether the documents are improved. In thisinstance, improved includes audited plus document formation issystematically rooted in most, if not all, aspects of the system.

If the documents are not improve, the method continues at step 744 wherethe analysis system generates a documentation rating of 15 (and/or aword rating of “metric & reporting”). If the documents are improve, themethod continues at step 745 where the analysis system generates adocumentation rating of 20 (and/or a word rating of “improvement”). Notethat the numerical ratings are example values and could be other values.Further note that the number of level of documentation rating may bemore or less than the five shown.

For this method, distinguishing between formalized, metric & reporting,and improvement is interpretative based on the manner in which the data600 was analyzed. As an example, weighting factors on certain types ofanalysis affect the level. As a specific example, weighting factors foranalysis to determine last revisions of documents, age of lastrevisions, content verification of documents with respect to achecklist, balance of local documents and system-wide documents, topicverification of the documents with respect to desired topics, and/ordocument language evaluation will affect the resulting level.

FIG. 100 is a logic diagram of an example of generating a documentationrating by the analysis system; in particular the documentation ratingmodule generating a documentation rating based on use of documents. Themethod begins at step 746 where the analysis system determines whetherthere is at least one use of a document. Note that the threshold numberin this step could be greater than one. If there are no use ofdocuments, the method continues at step 747 where the analysis systemgenerates a documentation rating of 0 (and/or a word rating of “none”).

If there is at least one use of a document, the method continues at step748 where the analysis system determines whether the use of thedocuments is formalized. In this instance, formalized use of documentsinclude sufficient detail regarding how to use the documentation,include use variations from document to document, use of documents isnot routinely reviewed in an organized manner, and/or use of documentsis not regulated.

If the use of documents is not formalized, the method continues at step749 where the analysis system generates a documentation rating of 5(and/or a word rating of “informal”). If, however, the use of documentsis at least formalized, the method continues at step 750 where theanalysis system determines whether the use of the documents is metric &reporting. In this instance, metric & reporting includes formal plus useof documents is routinely reviewed, and/or the use of documents isregulated.

If the use of documents is not metric & reporting, the method continuesat step 751 where the analysis system generates a documentation ratingof 10 (and/or a word rating of “formal”). If, however, the use ofdocuments is at least metric & reporting, the method continues at step752 where the analysis system determines whether the use of documents isimproved. In this instance, improved includes metric & reporting plususe of document is systematically rooted in most, if not all, aspects ofthe system.

If the use of documents is not improve, the method continues at step 753where the analysis system generates a documentation rating of 15 (and/ora word rating of “metric & reporting”). If the use of documents isimprove, the method continues at step 754 where the analysis systemgenerates a documentation rating of 20 (and/or a word rating of“improvement”). Note that the numerical ratings are example values andcould be other values. Further note that the number of level ofdocumentation rating may be more or less than the five shown.

FIG. 101 is a logic diagram of an example of generating a documentationrating by the analysis system; in particular the documentation ratingmodule generating a documentation rating based on application ofdocuments. The method begins at step 755 where the analysis systemdetermines whether there is at least one application of a document. Notethat the threshold number in this step could be greater than one. Ifthere are no applications of documents, the method continues at step 756where the analysis system generates a documentation rating of 0 (and/ora word rating of “none”).

If there is at least one application of a document, the method continuesat step 757 where the analysis system determines whether the applicationof the documents is formalized. In this instance, formalized applicationof documents include sufficient detail regarding how to apply thedocumentation, include application variations from document to document,application of documents is not routinely reviewed in an organizedmanner, and/or application of documents is not regulated.

If the application of documents is not formalized, the method continuesat step 758 where the analysis system generates a documentation ratingof 5 (and/or a word rating of “informal”). If, however, the applicationof documents is at least formalized, the method continues at step 759where the analysis system determines whether the application of thedocuments is metric & reporting. In this instance, metric & reportingincludes formal plus application of documents is routinely reviewed,and/or the application of documents is regulated.

If the application of documents is not metric & reporting, the methodcontinues at step 760 where the analysis system generates adocumentation rating of 10 (and/or a word rating of “formal”). If,however, the application of documents is at least metric & reporting,the method continues at step 761 where the analysis system determineswhether the application of documents is improve. In this instance,improve includes metric & reporting plus use of document issystematically rooted in most, if not all, aspects of the system.

If the application of documents is not improve, the method continues atstep 762 where the analysis system generates a documentation rating of15 (and/or a word rating of “metric & reporting”). If the application ofdocuments is improve, the method continues at step 763 where theanalysis system generates a documentation rating of 20 (and/or a wordrating of “improvement”). Note that the numerical ratings are examplevalues and could be other values. Further note that the number of levelof documentation may be more or less than the five shown.

FIG. 102 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular generating an automation rating. The method begins at step764 where the analysis system generates a first automation rating basedon a first combination of a system criteria (e.g., system requirements),of a system mode (e.g., system functions), of an evaluation perspective(e.g., implementation), and of an evaluation viewpoint (e.g., discloseddata).

The method continues at step 765 where the analysis system generates asecond automation rating based on a second combination of a systemcriteria (e.g., system design), of a system mode (e.g., systemfunctions), of an evaluation perspective (e.g., implementation), and ofan evaluation viewpoint (e.g., disclosed data). The method continues atstep 766 where the analysis system generates the automation rating basedon the first and second automation ratings.

FIG. 103 is a logic diagram of a further example of generating anautomation rating for understanding of system build of issue detectionsecurity functions of an organization. The method begins at step 767where the analysis system identifies automation regarding issuedetection security functions from the data. The method continues at step768 where the analysis system generates an automation rating from thedata. Examples of this were discussed with reference to FIG. 81.

The method also continues at step 769 where the analysis systemdetermines use of the automation for issue detection security functions.The method continues at step 770 where the analysis system generates anautomation rating based on use. Examples of this were discussed withreference to FIG. 81.

The method also continues at step 771 where the analysis systemdetermines consistency of applying the automation for issue detectionsecurity functions. The method continues at step 772 where the analysissystem generates an automation rating based on consistency of use.Examples of this were discussed with reference to FIG. 81. The methodcontinues at step 773 where the analysis system generates the automationrating based on the automation rating from the data, the automationrating based on use, and the automation rating based on application(i.e., consistency of use).

FIG. 104 is a logic diagram of a further example of generating anautomation rating for understanding of system build of issue detectionsecurity functions of an organization. The method begins at step 774where the analysis system identifies automation to verify issuedetection security functions from the data. The method continues at step775 where the analysis system generates an automation rating from thedata. Examples of this were discussed with reference to FIG. 81.

The method also continues at step 776 where the analysis systemdetermines use of the automation for issue detection security functions.The method continues at step 777 where the analysis system generates anautomation rating based on use of the verify automation. Examples ofthis were discussed with reference to FIG. 81.

The method also continues at step 778 where the analysis systemdetermines consistency of applying the verifying automation for issuedetection security functions. The method continues at step 779 where theanalysis system generates an automation rating based on consistency ofuse. Examples of this were discussed with reference to FIG. 81. Themethod continues at step 780 where the analysis system generates theautomation rating based on the automation rating from the data, theautomation rating based on use, and the automation rating based onconsistency of use.

FIG. 105 is a logic diagram of an example of generating an automationrating by the analysis system; in particular the automation ratingmodule generating an automation rating based on data 600. The methodbegins at step 781 where the analysis system determines whether there isavailable automation for a particular system aspect, system criteria,and/or system mode. If automation is not available, the method continuesat step 782 where the analysis system generates an automation rating of10 (and/or a word rating of “unavailable”).

If automation is available, the method continues at step 783 where theanalysis system determines whether there is at least one automation inthe data. If not, the method continues at step 784 where the analysissystem generates an automation rating of 0 (and/or a word rating of“none”).

If there is at least one automation, the method continues at step 785where the analysis system determines whether full automation is found inthe data. In this instance, full automation refers to the automationtechniques that are available for the system are in the data 600.

If the automation is not full, the method continues at step 786 wherethe analysis system generates an automation rating of 5 (and/or a wordrating of “partial”). If, however, the automation is full, the methodcontinues at step 787 where the analysis system generates an automationrating of 10 (and/or a word rating of “full”). Note that the numericalratings are example values and could be other values. Further note thatthe number of level of automation may be more or less than the fourshown.

FIG. 106 is a logic diagram of an example of generating an automationrating by the analysis system; in particular the automation ratingmodule generating an automation rating based on use. The method beginsat step 788 where the analysis system determines whether there isavailable automation for a particular system aspect, system criteria,and/or system mode. If automation is not available, the method continuesat step 789 where the analysis system generates an automation rating of10 (and/or a word rating of “unavailable”).

If automation is available, the method continues at step 790 where theanalysis system determines whether there is at least one use ofautomation. If not, the method continues at step 791 where the analysissystem generates an automation rating of 0 (and/or a word rating of“none”).

If there is at least one use of automation, the method continues at step792 where the analysis system determines whether automation is fullyused. In this instance, full use of automation refers to the automationtechniques that the system has are fully used.

If the use of automation is not full, the method continues at step 793where the analysis system generates an automation rating of 5 (and/or aword rating of “partial”). If, however, the use of automation is full,the method continues at step 794 where the analysis system generates anautomation rating of 10 (and/or a word rating of “full”).

FIG. 107 is a logic diagram of an example of generating an automationrating by the analysis system; in particular the automation ratingmodule generating an automation rating based on application ofautomation. The method begins at step 795 where the analysis systemdetermines whether there is available automation for a particular systemaspect, system criteria, and/or system mode. If automation is notavailable, the method continues at step 796 where the analysis systemgenerates an automation rating of 10 (and/or a word rating of“unavailable”).

If automation is available, the method continues at step 790 where theanalysis system determines whether there is at least one application ofautomation. If not, the method continues at step 798 where the analysissystem generates an automation rating of 0 (and/or a word rating of“none”).

If there is at least one application of automation, the method continuesat step 799 where the analysis system determines whether automation isfully applied. In this instance, full application of automation refersto the automation techniques of the system are applied to achieveconsistent use.

If the application of automation is not full, the method continues atstep 800 where the analysis system generates an automation rating of 5(and/or a word rating of “partial”). If, however, the application ofautomation is full, the method continues at step 801 where the analysissystem generates an automation rating of 10 (and/or a word rating of“full”).

FIG. 108 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof;in particular the analysis system identifying system elements for use inthe issue detection evaluation. The method begins at step 810 where theanalysis system activating at least one detection tool (e.g.,cybersecurity tool, end point tool, network tool, IP address tool,hardware detection tool, software detection tool, etc.) based on thesystem aspect.

The method continues at step 811 where the analysis system determineswhether an identified system element has already been identified for thesystem aspect (e.g., is already in the collection of data 600 and/or ispart of the gathered data). If yes, the method continues at step 812where the analysis system determines whether the identifying of systemelements is done. If not, the method repeats at step 811. If theidentifying of system elements is done, the method continues at step 813where the analysis system determines whether to end the method or repeatit for another system aspect, or portion thereof.

If, at step 811, the identified system is element is not included in thecollection of data, the method continues at step 814 where the analysissystem determines whether the potential system element is alreadyidentified as being a part of the system aspect, but not included in thecollection of data 600 (e.g., is it cataloged as being part of thesystem?). If yes, the method continues at step 815 where the analysissystem adds the identified system element to the collection of data 600.

If, at step 814, the system element is not cataloged as being part ofthe system, the method continues at step 816 where the analysis systemobtains data regarding the potential system element. For example, theanalysis system obtains a device ID, a user ID, a device serial number,a device description, a software ID, a software serial number, asoftware description, vendor information and/or other data regarding thesystem element.

The method continues at step 816 where the analysis system verifies thepotential system element based on the data. For example, the analysissystem verifies one or more of a device ID, a user ID, a device serialnumber, a device description, a software ID, a software serial number, asoftware description, vendor information and/or other data regarding thesystem element to establish that the system element is a part of thesystem. When the potential system element is verified, the methodcontinues at step 818 where the analysis system adds the system elementas a part of the system aspect (e.g., catalogs it as part of the systemand/or adds it to the collection of data 600).

FIG. 109 is a diagram of an example of system aspects, evaluationaspects, evaluation rating metrics, and analysis system output optionsof an analysis system 11 for analyzing a system 11, or portion thereof.For instance, analysis system 11 is evaluating, with respect to process,policy, procedure, certification, documentation, and/or automation, theunderstanding and implementation of the guidelines, system requirements,system design, and/or system build for issue detection securityfunctions of an organization based on disclosed data and discovered datato produce an evaluation rating.

For this example, the analysis system 10 can generate one or a pluralityof issue detection evaluation ratings for implementation of theguidelines, system requirements, system design, and/or system build forissue detection security functions of an organization based on discloseddata and discovered data in accordance with the evaluation ratingmetrics of process, policy, procedure, certification, documentation,and/or automation. A few, but far from exhaustive, examples are shown inFIGS. 113-117.

FIG. 110 is an example of issue detection data for use by an analysissystem to generate an issue detection rating for a system, or portionthereof. FIG. 110 is similar to FIG. 77 except, in addition to disclosedinformation regarding system build of issue detection security functionsof an organization, here, the issue detection information furtherincludes discovered information regarding system build, system design,and implementation of issue detection security functions of theorganization. Discovered information is the information discovered aboutthe system by the analysis system during the analysis.

In this example, the discovered and disclosed anomalies and eventawareness information is gathered from various disclosed and discovereddata sources as a portion of the discovered and disclosed issuedetection information recorded as issue detection data. The disclosedportion of the discovered and disclosed anomalies and event awarenessinformation may include information obtained from disclosed diagrams, alist of system tools, disclosed device information, disclosedapplication information, the system security plan, and disclosed reportsrelevant to anomalies and event awareness as discussed with reference toFIG. 77.

The discovered portion of the of the discovered and disclosed anomaliesand event awareness information is obtained from analyzing at least aportion of the disclosed information and from engagement with systemdevices, operating systems, networks, system tools, one or moredepartments (e.g., the IT department), one or more groups, one or moreusers (e.g., discovered user data), etc. For example, by engaging in anetwork analysis, the analysis system may discover devices (e.g.,discovered devices) connected to an organization's network that were notin the disclosed information.

By engaging with devices of the organization, the analysis system isoperable to analyze whether the devices have any hardware and/orsoftware issues. For example, hardware and/or software may be improperlyinstalled, old, and/or misused. As another example, hardware may notefficiently support software.

By engaging with various system tools, the analysis system can determinehow well the system tools are working and if they are executing theirintended purposes. For example, the analysis system can engage with thelog management tool, gather logs from system components, and determinelog management processes to assess the log management tool'scapabilities. As an example, the analysis system may discover that notall log data is being collected and additional log collection forwardingdevices (e.g., hardware and/or software) are needed. As another example,the analysis system may discover that while the log management toolaggregates event data from various sources, the event analysis componentof the log management tool is limited (e.g., the security analyst teamis short staffed and unable to manage the data). When the analysissystem outputs include deficiency identification and/or auto-correction,the analysis system is operable to recommend or auto-install a SIEM toolor similar event aggregation and analysis tool to remedy the issue.

By engaging with various departments, groups, users (e.g., discovereduser data such as user communications (e.g., email content, memos,internal documents, text messages, etc.)), etc., the analysis systemdevelops an understanding of system data flow, network operations,personnel roles and responsibilities, and execution of disclosedprocesses, policies, procedures, etc. For example, a disclosed issuedetection policy may require the security analyst team to alert thelegal department upon discovery of an incident. However, the analysissystem discovers that there is no clear definition of the term incidentin the disclosed information and the issue detection policy is flawed.As another example, the analysis system may discover that the incidentalert threshold is set too low and is creating noise and alarm fatigue.Numerous other pieces of discovered and disclosed anomalies and eventawareness information may be gathered from various information sources.The above examples are far from exhaustive.

FIG. 111 is an example of issue detection data for use by an analysissystem to generate an issue detection rating for a system, or portionthereof. FIG. 111 is similar to FIG. 78 except, in addition to disclosedinformation regarding system build of issue detection security functionsof an organization, here, the issue detection information furtherincludes discovered information regarding system build, system design,and implementation of issue detection security functions of theorganization. Discovered information is the information discovered aboutthe system by the analysis system during the analysis.

In this example, discovered and disclosed continuous security monitoringinformation is gathered from various data sources as a portion of thediscovered and disclosed issue detection information recorded as issuedetection data. The disclosed portion of the discovered and disclosedcontinuous security monitoring information may include the disclosedsystem security plan, disclosed list of system tools, disclosed deviceinformation, training materials, and disclosed user information relevantto continuous security monitoring as discussed with reference to FIG.78.

The discovered portion of the of the discovered and disclosed continuoussecurity monitoring information is obtained from analyzing at least aportion of the disclosed information and from engagement with the systemdevices, applications, networks, system tools, one or more departments(e.g., the IT department), one or more groups, one or more users (e.g.,discovered user data), etc.

For example, by engaging with the IT department, the analysis systemdevelops an understanding of security monitoring data flows, personnelroles and responsibilities, and execution of disclosed continuoussecurity monitoring processes, policies, procedures, etc.

By engaging in a network analysis, the analysis system may discoverdevices (e.g., discovered devices) connected to an organization'snetwork that were not in the disclosed information. By engaging withsystem devices, the analysis system is operable to analyze whether thedevices have any hardware and/or software issues. For example, adisclosed malicious code detection policy may state that all systemdevices require current antivirus software, however; the analysis systemanalyzes system devices to determine that several devices have noantivirus software installed or have an old version installed.

By engaging with various system tools, the analysis system can determinehow well the system tools are working and if they are executing theirintended purposes. For example, the analysis system can engage with thenetwork monitoring tool, gather network traffic data, and determinenetwork monitoring processes to assess the network monitoring tool'scapabilities. As an example, the analysis system may discover that animportant device is not being monitored. As another example, theanalysis system may discover that the network monitoring tool has anegative performance impact on the devices where it is implemented. Asanother example, the analysis system may discover that external serviceprovider activity is not monitored. When the analysis system outputsinclude deficiency identification and/or auto-correction, the analysissystem is operable to recommend and/or auto-install remedies such as anexternal service provider activity monitoring tool (e.g., a cloud accesssecurity broker (CASB)) in this instance.

As another example, the analysis system can assess the quality andperformance of an alarm system installed at a facility for purposes ofphysical environment monitoring. As a further example, the analysissystem can assess the antivirus and malware protection softwareimplemented by the system to determine whether it is appropriate for theorganization and has the necessary features (e.g., auto-update, centralmanagement, etc.).

By engaging with various groups, users (e.g., discovered user data suchas user communications (e.g., email content, memos, internal documents,text messages, etc.)), etc., the analysis system develops anunderstanding of system data flow, personnel roles and responsibilities,and execution of disclosed processes, policies, procedures, etc. Forexample, disclosed personnel training materials may disclose policiesregarding user uploads and network connections. By engaging with devicesand analyzing communications on the network, the analysis system canassess whether the users are adhering to the practices outlined in thetraining materials (e.g., are users performing unauthorized uploads, areusers connecting to unknown networks, are users falling victim tophishing emails, etc.). Numerous other pieces of discovered anddisclosed continuous security monitoring information may be gatheredfrom various information sources. The above examples are far fromexhaustive.

FIG. 112 is an example of issue detection data for use by an analysissystem to generate an issue detection rating for a system, or portionthereof. FIG. 112 is similar to FIG. 79 except, in addition to disclosedinformation regarding system build of issue detection security functionsof an organization, here, the issue detection information furtherincludes discovered information regarding system build, system design,and implementation of issue detection security functions of theorganization. Discovered information is the information discovered aboutthe system by the analysis system during the analysis.

In this example, discovered and disclosed issue detection processesanalysis information is gathered from various sources as a portion ofthe discovered and disclosed issue detection information recorded asissue detection data. The disclosed portion of the discovered anddisclosed issue detection processes analysis information may include thedisclosed system security plan, disclosed diagrams, a disclosed list ofsystem tools, personnel handbooks, human resources (HR) documents,personnel training materials, reports, and other disclosed internaldocuments as discussed with reference to FIG. 79.

The discovered portion of the of the discovered and disclosed issuedetection processes analysis information is obtained from analyzing atleast a portion of the disclosed information and from engagement withthe system devices, operating systems, networks, system tools, one ormore departments (e.g., the IT department), one or more groups, one ormore users (e.g., discovered user data), etc.

For example, by engaging with system devices and various system tools,the analysis system can determine how well the system tools are workingand if they are executing their intended purposes. For example, theanalysis system can engage with verification tools, gather network anddata flow information, and determine issue detection processes to assessthe verification tools' capabilities. As an example, the analysis systemmay discover that vulnerability scanning processes are insufficient forthe system or a portion thereof (e.g., scanning is too infrequent).

By engaging with system devices and various departments (e.g., the ITdepartment, the HR department, etc.), the analysis system develops anunderstanding of system data flow, personnel roles and responsibilities,and execution of disclosed processes, policies, procedures, etc. Forexample, the analysis system can analyze whether there is overlap in theroles and responsibilities of the security analyst team, whethersecurity reports are sent to the right parties, whether personnel act inaccordance with their roles, etc.

By analyzing disclosed personnel training materials and reports and byengaging with users of the system (e.g., discovered user data such asuser communications (e.g., email content, memos, internal documents,text messages, etc.)), the analysis system can assess whether trainingoccurs often enough, covers the correct topics, involves the rightpeople, and whether the matters discussed are enforced. Numerous otherpieces of discovered and disclosed issue detection processes analysisinformation may be gathered from various information sources. The aboveexamples are far from exhaustive.

FIG. 113 is a diagram of an example of producing a plurality of issuedetection ratings and combining them into one rating. In this example,sixteen individual ratings are generated and then, via the cumulativerating module 607, are combined into one issue detection rating 608. Afirst individual issue detection rating is generated from a combinationof organization, guidelines, security functions, understanding, anddisclosed data. A second individual issue detection rating is generatedfrom a combination of organization, guidelines, security functions,implementation, and disclosed data. A third individual issue detectionrating is generated from a combination of organization, guidelines,security functions, understanding, and discovered data. A fourth issuedetection issue detection rating is generated from a combination oforganization, guidelines, security functions, implementation, anddiscovered data. The remaining twelve individual issue detection ratingsare generated from the combinations shown.

FIG. 114 is a diagram of another example of producing a plurality ofissue detection ratings and combining them into one rating. In thisexample, two individual ratings are generated and then, via thecumulative rating module 607, are combined into one issue detectionrating 608. A first individual issue detection rating is generated froma combination of organization, guidelines, security functions,understanding, and disclosed data. A second individual issue detectionrating is generated from a combination of organization, guidelines,security functions, implementation, and disclosed data. This allows fora comparison between the understanding of the issue detection securityfunctions of the organization from the guidelines of the disclosed dataand the implementation of the issue detection security functions of theorganization from the guidelines of the disclosed data. This comparisonprovides a metric for determining how well the guidelines wereunderstood and how well they were used and/or applied.

FIG. 115 is a diagram of another example of producing a plurality ofissue detection ratings and combining them into one rating. In thisexample, two individual ratings are generated and then, via thecumulative rating module 607, are combined into one issue detectionrating 608. A first individual issue detection rating is generated froma combination of organization, guidelines, security functions,understanding, and disclosed data. A second individual issue detectionrating is generated from a combination of organization, guidelines,security functions, understanding, and discovered data. This allows fora comparison between the understanding of the issue detection securityfunctions of the organization from the guidelines of the disclosed dataand the understanding of the issue detection security functions of theorganization from the guidelines of the discovered data. This comparisonprovides a metric for determining how well the guidelines were believedto be understood and how well they were actually understood.

FIG. 116 is a diagram of another example of producing a plurality ofissue detection ratings and combining them into one rating. In thisexample, two individual ratings are generated and then, via thecumulative rating module 607, are combined into one issue detectionrating 608. A first individual issue detection rating is generated froma combination of organization, guidelines, security functions,understanding, and disclosed data. A second individual issue detectionrating is generated from a combination of organization, systemrequirements, security functions, understanding, and disclosed data.This allows for a comparison between the understanding of the issuedetection security functions of the organization from the guidelines ofthe disclosed data and the understanding of the issue detection securityfunctions of the organization from the system requirements of thedisclosed data. This comparison provides a metric for determining howwell the guidelines were converted into the system requirements.

FIG. 117 is a diagram of another example of producing a plurality ofissue detection ratings and combining them into one rating. In thisexample, four individual ratings are generated and then, via thecumulative rating module 607, are combined into one issue detectionrating 608. A first individual issue detection rating is generated froma combination of organization, guidelines, security functions,understanding, and disclosed data. A second individual issue detectionrating is generated from a combination of organization, systemrequirements, security functions, understanding, and disclosed data. Athird individual issue detection rating is generated from a combinationof organization, system design, security functions, understanding, anddisclosed data. A fourth individual issue detection rating is generatedfrom a combination of organization, system build, security functions,understanding, and disclosed data.

This allows for a comparison between the understanding of the issuedetection security functions of the organization from the guidelines ofthe disclosed data, the understanding of the issue detection securityfunctions of the organization from the system requirements of thedisclosed data, the understanding of the issue detection securityfunctions of the organization from the system design of the discloseddata, and the understanding of the issue detection security functions ofthe organization from the system build of the disclosed data. Thiscomparison provides a metric for determining how well the guidelines,system requirements, system design, and/or system build were understoodwith respect to each and how well they were used and/or applied.

FIG. 118 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof,and determining deficiencies. The method includes one or more of steps820-823. At step 820, the analysis system determines a system criteriadeficiency (e.g., guidelines, system requirements, system design, systembuild, and/or resulting system) of the system aspect based on the issuedetection rating and the issue detection data. Examples have beendiscussed with reference to one or more preceding figures.

At step 821, the analysis system determines a system mode deficiency(e.g., assets, system functions, and/or security functions) of thesystem aspect based on the issue detection rating and the issuedetection data. At step 822, the analysis system determines anevaluation perspective deficiency (e.g., understanding, implementation,operation, and/or self-analysis) of the system aspect based on the issuedetection rating and the issue detection data. At step 823, the analysissystem determines an evaluation viewpoint deficiency (e.g., disclosed,discovered, and/or desired) of the system aspect based on the issuedetection rating and the issue detection data. Examples have beendiscussed with reference to one or more preceding figures.

FIG. 119 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof,and determining auto-corrections. The method begins at step 824 wherethe analysis system determines a deficiency of the system aspect basedon the issue detection rating and/or the issue detection data asdiscussed with reference to FIG. 118. The method continues at step 825where the analysis system determines whether the deficiency isauto-correctable. For example, is the deficiency regarding software andif so, can it be auto-corrected.

If the deficiency is not auto-correctable, the method continues at step826 where the analysis system includes the identified deficiency in areport. If, however, the deficiency is auto-correctable, the methodcontinues at step 827 where the analysis system auto-corrects thedeficiency. The method continues at step 828 where the analysis systemincludes the identified deficiency and auto-correction in a report.Examples of auto-correction have been discussed with reference to one ormore preceding Figures.

FIG. 120 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a system, or portion thereof.The method begins at step 830 where the analysis system selects asystem, or portion thereof, to evaluate issue detection of the system,or portion thereof, with respect to assets, system functions, and/orsecurity functions. For example, the analysis system selects one or moresystem elements, one or more system criteria, and/or one or more systemmodes for the system, or portion thereof, to be evaluated. As anotherexample, the analysis system selects one more evaluation perspective,one or more evaluation viewpoints, and/or detect as the for theevaluation category. The analysis system may further select one or moredetect sub-categories and/or one or more sub-sub categories.

As another example of selecting the system or portion thereof, theanalysis system selects the entire system (e.g., theorganization/enterprise), selects a division of an organizationoperating the system, selects a department of a division, selects agroup of a department, or selects a sub-group of a group. As anotherexample selecting the system or portion thereof, the analysis systemselects one or more physical assets and/or one or more conceptualassets.

The method continues at step 831 where the analysis system obtains issuedetection information regarding the system, or portion thereof. Theissue detection information includes information representative of anorganization's understanding of issue detection of the system, orportion thereof, with respect to the assets, the system functions,and/or the security functions. In an example, the analysis systemobtains the issue detection information (e.g., disclosed data from thesystem) by receiving it from a system admin computing entity. In anotherexample, the analysis system obtains the issue detection information bygathering it from one or more computing entities of the system.

The method continues at step 832 where the analysis system engages withthe system, or portion thereof, to produce system issue detection data(e.g., discovered data) regarding the system, or portion thereof, withrespect to the assets, the system functions, and/or the securityfunctions. Engaging the system, or portion thereof, will be discussed ingreater detail with reference to FIG. 121.

The method continues at step 833 where the analysis system calculates anissue detection rating regarding the issue detection of the system, orportion thereof, based on the issue detection information, the systemissue detection data, and issue detection processes, issue detectionpolicies, issue detection documentation, and/or issue detectionautomation. The issue detection rating may be indicative of a variety offactors of the system, or portion thereof. For example, the issuedetection rating indicates how well the issue detection informationreflects an understanding of issue detection with respect to securityfunctions of the system, or portion thereof. As another example, theissue detection rating indicates how well the issue detectioninformation reflects an understanding of issue detection with respect tosystem functions of system, or portion thereof.

As another example, the issue detection rating indicates how well theissue detection information reflects an understanding of issue detectionwith respect to assets the system, or portion thereof. As anotherexample, the issue detection rating indicates how well the issuedetection information reflects intended implementation of issuedetection with respect to assets of the system, or portion thereof. Asanother example, the issue detection rating indicates how well the issuedetection information reflects intended operation of issue detectionwith respect to assets of the system, or portion thereof.

As another example, the issue detection rating indicates how well theissue detection information reflects intended implementation of issuedetection with respect to system functions of the system, or portionthereof. As another example, the issue detection rating indicates howwell the issue detection information reflects intended operation ofissue detection with respect to system functions of system, or portionthereof.

As another example, the issue detection rating indicates how well theissue detection information reflects intended implementation of issuedetection with respect to security functions of the system, or portionthereof. As another example, the issue detection rating indicates howwell the issue detection information reflects intended operation ofissue detection with respect to security functions of the system, orportion thereof.

The method continues at step 834 where the analysis system gathersdesired system issue detection data from one or more system proficiencyresources. The method continues at step 835 where the analysis systemcalculates a second issue detection rating regarding a desired level ofissue detection of the system, or portion thereof, based on the issuedetection information, the issue detection data, the desired issuedetection data, and the issue detection processes, the issue detectionpolicies, the issue detection documentation, and/or the issue detectionautomation. The second issue detection rating is regarding a comparisonof desired data with the disclosed data and/or discovered data.

FIG. 121 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for system, or portion thereof; inparticular engaging the system, or portion thereof to obtain data. Themethod begins at step 836 where the analysis system interprets the issuedetection information to identify components (e.g., computing device,HW, SW, server, etc.) of the system, or portion thereof. The methodcontinues at step 837 where the analysis system queries a componentregarding implementation, function, and/or operation of the component.

The method continues at step 838 where the analysis system evaluates aresponse from the component concurrence with a portion of the issuedetection information relevant to the component. The method continues atstep 839 where the analysis system determines whether the responseconcurs with a portion of the issue detection information. If theresponse concurs, the method continues at step 840 where the analysissystem adds a data element (e.g., a record entry, a note, set a flag,etc.) to the system issue detection data regarding the substantialconcurrence of the response from the component with the portion of theissue detection information relevant to the component.

If the response does not concur, the method continues at step 841 wherethe analysis system adds a data element (e.g., a record entry in atable, a note, set a flag, etc.) to the system issue detection dataregarding the response from the component not substantially concurringwith the portion of the issue detection information relevant to thecomponent. The non-concurrence is indicative of a deviation in theimplementation, function, and/or operation of the component asidentified in the response from disclosed implementation, function,and/or operation of the component as contained in the issue detectioninformation. For example, the deviation is different HW, different SW,different network access, different data access, different data flow,coupled to different other components, and/or other differences.

The method continues in FIG. 122 at step 842 where the analysis systemqueries the component and/or another component regarding a cause for thedeviation. The method continues at step 843 where the analysis systemupdates a data element to include an indication of the one or morecauses for a deviation, wherein a cause for the deviation is based onresponses from the component and/or the other component.

The method continues at step 844 where the analysis system determineswhether the deviation is a communication deviation. If yes, the methodcontinues at step 845 where the analysis system evaluate a response fromthe device to ascertain an error of the issue detection informationregarding the device and/or the communication between the device and thecomponent. The method continues at step 846 where the analysis systemdetermines one or more causes of the error of the communicationdeviation.

If the deviation is not a communication deviation, the method continuesat step 847 where the analysis system determines whether the deviationis a system function deviation. If yes, the method continues at step 848where the analysis system evaluate a response from the device toascertain an error of the issue detection information regarding thedevice and/or the system function of the device. The method continues atstep 849 where the analysis system determines one or more causes of theerror of the system function deviation.

If the deviation is not a system function deviation, the methodcontinues at step 850 where the analysis system determines whether thedeviation is a security function deviation. If yes, the method continuesat step 851 where the analysis system evaluate a response from thedevice to ascertain an error of the issue detection informationregarding the device and/or the security function of the device. Themethod continues at step 852 where the analysis system determines one ormore causes of the error of the security function deviation.

If the deviation is not a security function deviation, the methodcontinues at step 853 where the analysis system evaluates a deviceresponse from the device to ascertain an error of the issue detectioninformation regarding the device and/or of the device. The methodcontinues at step 854 where the analysis system determines one or morecauses of the error of the information and/or of the device.

FIG. 123 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof,and in particular for calculating the issue detection rating. The methodbegins at steps 851 and 856. At step 851, the analysis system processesthe issue detection information into policy related issue detectioninformation, process related issue detection information, documentationrelated issue detection information, and/or automation related issuedetection information. At step 856, the analysis system processes thesystem issue detection data into policy related system issue detectiondata, process related system issue detection data, documentation relatedsystem issue detection data, and/or automation related system issuedetection data.

The method continues at steps 857-860. At step 857, the analysis systemevaluates the process related issue detection information with respectto the process related system issue detection data to produce a processissue detection rating. At step 858, the analysis system evaluates thepolicy related issue detection information with respect to the policyrelated system issue detection data to produce a policy issue detectionrating. At step 859, the analysis system evaluates the documentationrelated issue detection information with respect to the documentationrelated system issue detection data to produce a documentation issuedetection rating. At step 860, the analysis system evaluates theautomation related issue detection information with respect to theautomation related system issue detection data to produce an automationissue detection rating.

The method continues at step 861 where the analysis system generates anissue detection rating based on the automation issue detection rating,the documentation issue detection rating, the process issue detectionrating, and the policy issue detection rating. For example, the analysissystem performs a function on the automation issue detection rating, thedocumentation issue detection rating, the process issue detectionrating, and the policy issue detection rating to produce the issuedetection rating. The function is a weight average, standard deviation,statistical analysis, trending, and/or other mathematical function.

FIG. 124 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof,and in particular engaging the system to obtain issue detection data.The method begins at step 862 where the analysis system determines datagathering criteria and/or parameters. The determining of data gatheringparameters has been discussed with reference to one or more precedingFigures and one or more subsequent Figures.

The method continues at step 863 where the analysis system identifies auser device and queries it for data in accordance with the datagathering parameters. The method continues at step 864 where theanalysis system obtains a data response from the user device. The dataresponse includes data regarding the user device relevant to the datagathering parameters. An example of user device data was discussed withreference to one or more of FIGS. 75-79.

The method continues at step 865 where the analysis system catalogs theuser device (e.g., records it as being part of the system, or portionthereof, if not already cataloged). The method continues at step 866where the analysis system identifies vendor information regarding theuser device. The method continues at step 867 where the analysis systemtags the data regarding the user device with the vendor information.This enables data to be sorted, searched, etc. based on vendorinformation.

The method continues at step 868 where the analysis system determineswhether data has been received from all relevant user devices. If not,the method repeats at step 863. If yes, the method continues at step 869where the analysis system identifies a storage device and queries it fordata in accordance with the data gathering parameters. The methodcontinues at step 870 where the analysis system obtains a data responsefrom the storage device. The data response includes data regarding thestorage device relevant to the data gathering parameters. An example ofstorage device data was discussed with reference to one or more of FIGS.75-79.

The method continues at step 871 where the analysis system catalogs thestorage device (e.g., records it as being part of the system, or portionthereof, if not already cataloged) when the storage device responds. Themethod continues at step 872 where the analysis system identifies vendorinformation regarding the storage device. The method continues at step873 where the analysis system tags the data regarding the storage devicewith the vendor information. The method continues at step 874 where theanalysis system determines whether data has been received from allrelevant storage devices. If not, the method repeats at step 869.

If yes, the method continues at step 875 where the analysis systemidentifies a server device and queries it for data in accordance withthe data gathering parameters. The method continues at step 876 wherethe analysis system obtains a data response from the server device. Thedata response includes data regarding the server device relevant to thedata gathering parameters. An example of server device data wasdiscussed with reference to one or more of FIGS. 75-79. The methodcontinues at step 877 where the analysis system catalogs the serverdevice (e.g., records it as being part of the system, or portionthereof, if not already cataloged) when the server device responds.

The method continues at step 878 where the analysis system identifiesvendor information regarding the server device. The method continues atstep 879 where the analysis system tags the data regarding the serverdevice with the vendor information. The method continues at step 880 ofFIG. 125 where the analysis system determines whether data has beenreceived from all relevant server devices. If not, the method repeats atstep 875.

If yes, the method continues at step 881 where the analysis systemidentifies a security device and queries it for data in accordance withthe data gathering parameters. The method continues at step 882 wherethe analysis system obtains a data response from the security device.The data response includes data regarding the security device relevantto the data gathering parameters. An example of security device data wasdiscussed with reference to one or more of FIGS. 75-79.

The method continues at step 883 where the analysis system catalogs thesecurity device (e.g., records it as being part of the system, orportion thereof, if not already cataloged) when the security deviceresponds. The method continues at step 884 where the analysis systemidentifies vendor information regarding the security device. The methodcontinues at step 885 where the analysis system tags the data regardingthe security device with the vendor information. The method continues atstep 886 where the analysis system determines whether data has beenreceived from all relevant security devices. If not, the method repeatsat step 881.

If yes, the method continues at step 887 where the analysis systemidentifies a security tool and queries it for data in accordance withthe data gathering parameters. The method continues at step 888 wherethe analysis system obtains a data response from the security tool. Thedata response includes data regarding the security tool relevant to thedata gathering parameters. An example of security tool data wasdiscussed with reference to one or more of FIGS. 75-79.

The method continues at step 889 where the analysis system catalogs thesecurity tool (e.g., records it as being part of the system, or portionthereof, if not already cataloged) when the security tool responds viahardware on which the tool operates.

The method continues at step 890 where the analysis system identifiesvendor information regarding the security tool. The method continues atstep 891 where the analysis system tags the data regarding the securitytool with the vendor information. The method continues at step 892 wherethe analysis system determines whether data has been received from allrelevant security tools. If not, the method repeats at step 887.

If yes, the method continues at step 893 where the analysis systemidentifies a network device and queries it for data in accordance withthe data gathering parameters. The method continues at step 894 wherethe analysis system obtains a data response from the network device. Thedata response includes data regarding the network device relevant to thedata gathering parameters. An example of network device data wasdiscussed with reference to one or more of FIGS. 75-79. The methodcontinues at step 895 where the analysis system catalogs the networkdevice (e.g., records it as being part of the system, or portionthereof, if not already cataloged) when the network device responds.

The method continues at step 896 where the analysis system identifiesvendor information regarding the network device. The method continues atstep 897 where the analysis system tags the data regarding the networkdevice with the vendor information. The method continues at step 898 ofFIG. 126 where the analysis system determines whether data has beenreceived from all relevant network devices. If not, the method repeatsat step 893.

If yes, the method continues at step 899 where the analysis systemidentifies another device (e.g., any other device that is part of thesystem, interfaces with the system, uses the system, and/or supports thesystem) and queries it for data in accordance with the data gatheringparameters. The method continues at step 900 where the analysis systemobtains a data response from the other device. The data responseincludes data regarding the other device relevant to the data gatheringparameters. An example of other device data was discussed with referenceto one or more of FIGS. 75-79.

The method continues at step 901 where the analysis system catalogs theother device (e.g., records it as being part of the system, or portionthereof, if not already cataloged) when the other device responds. Themethod continues at step 902 where the analysis system identifies vendorinformation regarding the other device. The method continues at step 903where the analysis system tags the data regarding the other device withthe vendor information. The method continues at step 904 where theanalysis system determines whether data has been received from allrelevant other devices. If not, the method repeats at step 899.

If yes, the method continues at step 905 where the analysis systemidentifies another tool (e.g., any other tool that is part of thesystem, interprets the system, monitors the system, and/or supports thesystem) and queries it for data in accordance with the data gatheringparameters. The method continues at step 906 where the analysis systemobtains a data response from the other tool. The data response includesdata regarding the other tool relevant to the data gathering parameters.An example of other tool data was discussed with reference to one ormore of FIGS. 75-79. The method continues at step 907 where the analysissystem catalogs the other tool (e.g., records it as being part of thesystem, or portion thereof, if not already cataloged) when the othertool responds via hardware on which the tool operates

The method continues at step 908 where the analysis system identifiesvendor information regarding the other tool. The method continues atstep 909 where the analysis system tags the data regarding the othertool with the vendor information. The method continues at step 910 wherethe analysis system determines whether data has been received from allrelevant other tools. If not, the method repeats at step 905. If yes,the method continues at step 911 where the analysis system ends theprocess or repeats it for another part of the system.

FIG. 127 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereofand, in particular, identifying a device or a tool relevant to issuedetection. The method begins at step 920 where the analysis systemdetermines whether a device (e.g., hardware and/or software) or tool isalready included in the regarding the data gathering parametersinformation (e.g., the disclosed data for a particular analysis of thesystem, or portion thereof). If yes, the method continues at step 921where the analysis module determines whether it's done with identifyingdevices or tools relevant to issue detection. If yes, the method isended. If not, the method repeats at step 920.

If the device or tool is not in the issue detection information, themethod continues at step 922 where the analysis system engages one ormore detection (or discovery) tools to detect a device and/or a toolrelevant to issue detection. Examples of detection tools were discussedwith reference to one or more preceding figures. The method continues atstep 923 where the analysis system determines whether the detectiontool(s) has identified a device (e.g., hardware and/or software). Ifnot, the method continues at step 924 where the analysis systemdetermines whether the detection tool(s) has identified a tool. If not,the method repeats at step 921.

If a tool is identified, the method continues at step 925 where theanalysis system obtains a data response from the tool, via hardware onwhich the tool operates, in regard to a data gathering request. The dataresponse includes data regarding the tool. Examples of the dataregarding the tool were discussed with reference to one or more of FIGS.75-79.

The method continues with step 926 where the analysis system determineswhether the tool is cataloged (e.g., is part of the system, but is notincluded in the issue detection information for this particularevaluation). If yes, the method continues at step 926 where the analysissystem adds the tool to the issue detection information and the methodcontinues at step 921.

If the tool is not cataloged, the method continues at step 928 where theanalysis system verifies the tool as being part of the system and thencatalogs it as part of the system. The method continues at step 929where the analysis system identifies vendor information regarding thetool. The method continues at step 930 where the analysis system tagsthe data regarding the tool with the vendor information. The methodrepeats at step 921.

If, at step 923, a device is identified, the method continues at step931 where the analysis system obtains a data response from the device inregard to a data gathering request. The data response includes dataregarding the device. Examples of the data regarding the device werediscussed with reference to one or more of FIGS. 75-79.

The method continues with step 933 where the analysis system determineswhether the device (e.g., hardware and/or software) is cataloged (e.g.,is part of the system, but is not included in the issue detectioninformation for this particular evaluation). If yes, the methodcontinues at step 932 where the analysis system adds the devices to theissue detection information and the method continues at step 921.

If the device is not cataloged, the method continues at step 934 wherethe analysis system verifies the device as being part of the system andthen catalogs it as part of the system. The method continues at step 935where the analysis system identifies vendor information regarding thedevice. The method continues at step 936 where the analysis system tagsthe data regarding the device with the vendor information. The methodrepeats at step 921.

FIG. 128 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereofand, in particular, identifying a device or a tool relevant to issuedetection. The method begins at step 940 where the analysis systemdetermines whether a device (e.g., hardware and/or software) or tool isalready included in the issue detection information (e.g., the discloseddata for a particular analysis of the system, or portion thereof). Ifyes, the method continues at step 941 where the analysis moduledetermines whether it's done with identifying devices or tools. If yes,the method is ended. If not, the method repeats at step 940.

If the device or tool is not in the issue detection information, themethod continues at step 942 where the analysis system interprets datafrom an identified device and/or tool (e.g., already in the issuedetection information) with regards to a device or tool. For example,the analysis system looks for data regarding an identified deviceexchanging data with the device being reviewed. As another example, theanalysis system looks for data regarding a tool being used on the deviceunder review to repair a software issue.

The method continues at step 943 where the analysis system determineswhether the data has identified such a device (e.g., hardware and/orsoftware). If not, the method continues at step 944 where the analysissystem determines whether the detection tool(s) has identified such atool. If not, the method repeats at step 941.

If a tool is identified, the method continues at step 945 where theanalysis system obtains a data response from the tool, via hardware onwhich the tool operates, in regard to a data gathering request. The dataresponse includes data regarding the tool. Examples of the dataregarding the tool were discussed with reference to one or more of FIGS.75-79.

The method continues at step 947 where the analysis system determineswhether the tool is cataloged (e.g., is part of the system, but is notincluded in the issue detection information for this particularevaluation). If yes, the method continues at step 946 where the analysissystem adds the tool to the issue detection information and the methodcontinues at step 941.

If the tool is not cataloged, the method continues at step 948 where theanalysis system verifies the tool as being part of the system and thencatalogs it as part of the system. The method continues at step 949where the analysis system identifies vendor information regarding thetool. The method continues at step 950 where the analysis system tagsthe data regarding the tool with the vendor information. The methodrepeats at step 921.

If, at step 943, a device is identified, the method continues at step951 where the analysis system obtains a data response from the device inregard to a data gathering request. The data response includes dataregarding the device. Examples of the data regarding the device werediscussed with reference to one or more of FIGS. 75-79.

The method continues with step 953 where the analysis system determineswhether the device (e.g., hardware and/or software) is cataloged (e.g.,is part of the system, but is not included in the issue detectioninformation for this particular evaluation). If yes, the methodcontinues at step 952 where the analysis system adds the devices to theissue detection information and the method continues at step 941.

If the device is not cataloged, the method continues at step 954 wherethe analysis system verifies the device as being part of the system andthen catalogs it as part of the system. The method continues at step 955where the analysis system identifies vendor information regarding thedevice. The method continues at step 956 where the analysis system tagsthe data regarding the device with the vendor information. The methodrepeats at step 941.

FIG. 129 is a logic diagram of a further example of an analysis systemdetermining an issue detection rating for a system, or portion thereof,and in particular to generating data gathering criteria (or parameters).The method begins at step 960 where the analysis system determineswhether the current analysis is for the entire system or a portionthereof. If the analysis is for the entire system, the method continuesat step 962 where the analysis system prepares to analyze the entiresystem. If the analysis is for a portion of the system, the methodcontinues at step 961 where the analysis system determines theparticular section (e.g., identifies one or more system elements).

The method continues at step 963 where the analysis system determineswhether the current analysis has identified evaluation criteria (e.g.,guidelines, system requirements, system design, system build, and/orresulting system). If yes, the method continues at step 964 where theanalysis system determines the specific evaluation criteria. If not, themethod continues at step 965 where the analysis system determines a setof default evaluation criteria (e.g., one or more of the evaluationcriteria).

The method continues at step 966 where the analysis system determineswhether the current analysis has identified an evaluation mode (e.g.,assets, system functions, and/or security functions). If yes, the methodcontinues at step 966 where the analysis system determines the specificevaluation mode(s). If not, the method continues at step 967 where theanalysis system determines a set of default evaluation modes (e.g., oneor more of the evaluation modes).

The method continues at step 968 where the analysis system determineswhether the current analysis has identified an evaluation perspective(e.g., understanding, implementation, and/or operation). If yes, themethod continues at step 969 where the analysis system determines thespecific evaluation perspective(s). If not, the method continues at step970 where the analysis system determines a set of default evaluationperspectives (e.g., one or more of the evaluation perspectives).

The method continues at step 971 where the analysis system determineswhether the current analysis has identified an evaluation viewpoint(e.g., disclosed, discovered, desired, and/or self-analysis). If yes,the method continues at step 972 where the analysis system determinesthe specific evaluation viewpoint(s). If not, the method continues atstep 973 where the analysis system determines a set of defaultevaluation viewpoints (e.g., one or more of the evaluation viewpoints).

The method continues at step 974 where the analysis system determineswhether the current analysis has identified an evaluation category,and/or sub-categories (e.g., categories include identify, protect,detect, response, and/or recover). If yes, the method continues at step975 where the analysis system determines one or more specific evaluationcategories and/or sub-categories. If not, the method continues at step977 where the analysis system determines a set of default evaluationcategories and/or sub-categories (e.g., one or more of the evaluationcategories and/or sub-categories). The method continues at step 976where the analysis system determines the data gathering criteria (orparameters) based on the determination made in the previous steps.

FIG. 130 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a system, or portion thereof.The method begins at step 1000 where the analysis system selects asystem, or portion thereof, to evaluate anomaly and event awareness ofthe system, or portion thereof. For example, the analysis system selectsone or more system elements, one or more system criteria, and/or one ormore system modes for the system, or portion thereof, to be evaluated.As another example, the analysis system selects one more evaluationperspective and/or one or more evaluation viewpoints. The analysissystem may further select one or more sub-sub categories of the anomalyand event awareness sub-category of the issue detection (e.g., detect)category.

As another example of selecting the system or portion thereof, theanalysis system selects the entire system, selects a division of anorganization operating the system, selects a department of a division,selects a group of a department, or selects a sub-group of a group. Asanother example of selecting the system or portion thereof, the analysissystem selects one or more physical assets and/or one or more conceptualassets.

The method continues at step 1001 where the analysis system obtainsanomaly and event awareness information regarding the system, or portionthereof. The anomaly and event awareness information includesinformation representative of an organization's understanding of thesystem, or portion thereof, with respect to establishment and managementof network operations and expected data flows, detected event analysis,event data aggregation and correlation, event impact determination, andincident alert threshold establishment. In an example, the analysissystem obtains the anomaly and event awareness information (e.g.,disclosed data from the system) by receiving it from a system admincomputing entity. In another example, the analysis system obtains theanomaly and event awareness information by gathering it from one or morecomputing entities of the system.

The method continues at step 1002 where the analysis system engages withthe system, or portion thereof, to produce system anomaly and eventawareness data (e.g., discovered data) regarding the system, or portionthereof, with respect to the establishment and management of networkoperations and expected data flows, the detected event analysis, theevent data aggregation and correlation, the event impact determination,and the incident alert threshold establishment. Engaging the system, orportion thereof, was discussed with reference to FIG. 121.

The method continues at step 1003 where the analysis system calculatesan anomaly and event awareness rating regarding the anomaly and eventawareness of the system, or portion thereof, based on the anomaly andevent awareness information, the system anomaly and event awarenessdata, and anomaly and event awareness processes, anomaly and eventawareness policies, anomaly and event awareness documentation, and/oranomaly and event awareness automation. The anomaly and event awarenessrating may be indicative of a variety of factors of the system, orportion thereof. For example, the anomaly and event awareness ratingindicates how well the anomaly and event awareness information reflectsan understanding of the anomaly and event awareness with respect toassets of the system, or portion thereof. As another example, theanomaly and event awareness rating indicates how well the anomaly andevent awareness information reflects an understanding of the anomaly andevent awareness with respect to system functions of system, or portionthereof.

As another example, the anomaly and event awareness rating indicates howwell the anomaly and event awareness information reflects anunderstanding of the anomaly and event awareness with respect to thesecurity functions of the system, or portion thereof. As anotherexample, the anomaly and event awareness rating indicates how well theanomaly and event awareness information reflects intended implementationof the anomaly and event awareness with respect to the assets of thesystem, or portion thereof. As another example, the anomaly and eventawareness rating indicates how well the anomaly and event awarenessinformation reflects intended operation of the anomaly and eventawareness with respect to the assets of the system, or portion thereof.

As another example, the anomaly and event awareness rating indicates howwell the anomaly and event awareness information reflects intendedimplementation of the anomaly and event awareness with respect to thesystem functions of the system, or portion thereof. As another example,the anomaly and event awareness rating indicates how well the anomalyand event awareness information reflects intended operation of theanomaly and event awareness with respect to the system functions ofsystem, or portion thereof.

As another example, the anomaly and event awareness rating indicates howwell the anomaly and event awareness information reflects intendedimplementation of the anomaly and event awareness with respect to thesecurity functions of the system, or portion thereof. As anotherexample, the anomaly and event awareness rating indicates how well theanomaly and event awareness information reflects intended operation ofthe anomaly and event awareness with respect to the security functionsof the system, or portion thereof.

The method continues at step 1004 where the analysis system gathersdesired system anomaly and event awareness data from one or more systemproficiency resources. The method continues at step 1005 where theanalysis system calculates a second anomaly and event awareness ratingregarding a desired level of anomaly and event awareness of the system,or portion thereof, based on the anomaly and event awarenessinformation, the system anomaly and event awareness data, the desiredanomaly and event awareness data, and the anomaly and event awarenessprocesses, the anomaly and event awareness policies, the anomaly andevent awareness documentation, and/or the anomaly and event awarenessautomation. The second anomaly and event awareness rating is regarding acomparison of desired data with the disclosed data and/or discovereddata.

FIG. 131 is a logic diagram of an example of an analysis systemdetermining an anomaly and event awareness evaluation rating for asystem, or portion thereof. The method begins at step 1006 where theanalysis system determines a system aspect (see FIG. 69) of a system foran anomaly and event awareness evaluation. An anomaly and eventawareness evaluation includes evaluating the system's establishment andmanagement of network operations and expected data flows, detected eventanalysis, event data aggregation and correlation, event impactdetermination, and incident alert threshold establishment.

The method continues at step 1007 where the analysis system determinesat least one evaluation perspective for use in performing the anomalyand event awareness evaluation on the system aspect. An evaluationperspective is an understanding perspective, an implementationperspective, an operation perspective, or a self-analysis perspective.An understanding perspective is with regard to how well the assets,system functions, and/or security functions with respect to anomaly andevent awareness are understood. An implementation perspective is withregard to how well the assets, system functions, and/or securityfunctions with respect to anomaly and event awareness are implemented.An operation perspective is with regard to how well the assets, systemfunctions, and/or security functions with respect to anomaly and eventawareness operate. A self-analysis (or self-evaluation) perspective iswith regard to how well the system self-evaluates the understanding,implementation, and/or operation of assets, system functions, and/orsecurity functions with respect to anomaly and event awareness.

The method continues at step 1008 where the analysis system determinesat least one evaluation viewpoint for use in performing the anomaly andevent awareness evaluation on the system aspect. An evaluation viewpointis disclosed viewpoint, a discovered viewpoint, or a desired viewpoint.A disclosed viewpoint is with regard to analyzing the system aspectbased on the disclosed data. A discovered viewpoint is with regard toanalyzing the system aspect based on the discovered data. A desiredviewpoint is with regard to analyzing the system aspect based on thedesired data.

The method continues at step 1009 where the analysis system obtainsanomaly and event awareness data regarding the system aspect inaccordance with the at least one evaluation perspective and the at leastone evaluation viewpoint. Anomaly and event awareness data is dataobtained that is regarding the system aspect.

The method continues at step 1010 where the analysis system calculatesan anomaly and event awareness rating as a measure of system anomaly andevent awareness maturity for the system aspect based on the anomaly andevent awareness data, the at least one evaluation perspective, the atleast one evaluation viewpoint, and at least one evaluation ratingmetric. An evaluation rating metric is a process rating metric, a policyrating metric, a procedure rating metric, a certification rating, adocumentation rating metric, or an automation rating metric.

FIG. 132 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a system, or portion thereof.The method begins at step 1020 where the analysis system selects asystem, or portion thereof, to evaluate continuous security monitoringof the system, or portion thereof. For example, the analysis systemselects one or more system elements, one or more system criteria, and/orone or more system modes for the system, or portion thereof, to beevaluated. As another example, the analysis system selects one moreevaluation perspective and/or one or more evaluation viewpoints. Theanalysis system may further select one or more sub-sub categories of thecontinuous security monitoring sub-category of the issue detection(e.g., detect) category.

As another example of selecting the system or portion thereof, theanalysis system selects the entire system, selects a division of anorganization operating the system, selects a department of a division,selects a group of a department, or selects a sub-group of a group. Asanother example of selecting the system or portion thereof, the analysissystem selects one or more physical assets and/or one or more conceptualassets.

The method continues at step 1021 where the analysis system obtainscontinuous security monitoring information regarding the system, orportion thereof. The continuous security monitoring information includesinformation representative of an organization's understanding of thesystem, or portion thereof, with respect to network monitoring, physicalenvironment monitoring, personnel activity monitoring, malicious codedetection, unauthorized mobile code detection, external service provideractivity monitoring, unauthorized personnel monitoring, unauthorizedconnection monitoring, unauthorized device monitoring, unauthorizedsoftware monitoring, and vulnerability scanning.

In an example, the analysis system obtains the continuous securitymonitoring information (e.g., disclosed data from the system) byreceiving it from a system admin computing entity. In another example,the analysis system obtains the continuous security monitoringinformation by gathering it from one or more computing entities of thesystem.

The method continues at step 1022 where the analysis system engages withthe system, or portion thereof, to produce system continuous securitymonitoring data (e.g., discovered data) regarding the system, or portionthereof, with respect to the network monitoring, the physicalenvironment monitoring, the personnel activity monitoring, the maliciouscode detection, the unauthorized mobile code detection, the externalservice provider activity monitoring, the unauthorized personnelmonitoring, the unauthorized connection monitoring, the unauthorizeddevice monitoring, the unauthorized software monitoring, and thevulnerability scanning. Engaging the system, or portion thereof, wasdiscussed with reference to FIG. 121.

The method continues at step 1023 where the analysis system calculates acontinuous security monitoring rating regarding the continuous securitymonitoring of the system, or portion thereof, based on the continuoussecurity monitoring information, the system continuous securitymonitoring data, and continuous security monitoring processes,continuous security monitoring policies, continuous security monitoringdocumentation, and/or continuous security monitoring automation.

The continuous security monitoring rating may be indicative of a varietyof factors of the system, or portion thereof. For example, thecontinuous security monitoring rating indicates how well the continuoussecurity monitoring information reflects an understanding of thecontinuous security monitoring with respect to assets of the system, orportion thereof. As another example, the continuous security monitoringrating indicates how well the continuous security monitoring informationreflects an understanding of the continuous security monitoring withrespect to system functions of system, or portion thereof.

As another example, the continuous security monitoring rating indicateshow well the continuous security monitoring information reflects anunderstanding of the continuous security monitoring with respect to thesecurity functions of the system, or portion thereof. As anotherexample, the continuous security monitoring rating indicates how wellthe continuous security monitoring information reflects intendedimplementation of the continuous security monitoring with respect to theassets of the system, or portion thereof. As another example, thecontinuous security monitoring rating indicates how well the continuoussecurity monitoring information reflects intended operation of thecontinuous security monitoring with respect to the assets of the system,or portion thereof.

As another example, the continuous security monitoring rating indicateshow well the continuous security monitoring information reflectsintended implementation of the continuous security monitoring withrespect to the system functions of the system, or portion thereof. Asanother example, the continuous security monitoring rating indicates howwell the continuous security monitoring information reflects intendedoperation of the continuous security monitoring with respect to thesystem functions of system, or portion thereof.

As another example, the continuous security monitoring rating indicateshow well the continuous security monitoring information reflectsintended implementation of the continuous security monitoring withrespect to the security functions of the system, or portion thereof. Asanother example, the continuous security monitoring rating indicates howwell the continuous security monitoring information reflects intendedoperation of the continuous security monitoring with respect to thesecurity functions of the system, or portion thereof.

The method continues at step 1024 where the analysis system gathersdesired continuous security monitoring data from one or more systemproficiency resources. The method continues at step 1025 where theanalysis system calculates a second continuous security monitoringrating regarding a desired level of continuous security monitoring ofthe system, or portion thereof, based on the continuous securitymonitoring information, the system continuous security monitoring data,the desired continuous security monitoring data, and the continuoussecurity monitoring processes, the continuous security monitoringpolicies, the continuous security monitoring documentation, and/or thecontinuous security monitoring automation. The continuous securitymonitoring rating is regarding a comparison of desired data with thedisclosed data and/or discovered data.

FIG. 133 is a logic diagram of an example of an analysis systemdetermining a continuous security monitoring evaluation rating for asystem, or portion thereof. The method begins at step 1026 where theanalysis system determines a system aspect (see FIG. 69) of a system fora continuous security monitoring evaluation. A continuous securitymonitoring evaluation includes evaluating the system's networkmonitoring, physical environment monitoring, personnel activitymonitoring, malicious code detection, unauthorized mobile codedetection, external service provider activity monitoring, unauthorizedpersonnel monitoring, unauthorized connection monitoring, unauthorizeddevice monitoring, unauthorized software monitoring, and vulnerabilityscanning.

The method continues at step 1027 where the analysis system determinesat least one evaluation perspective for use in performing the continuoussecurity monitoring evaluation on the system aspect. An evaluationperspective is an understanding perspective, an implementationperspective, an operation perspective, or a self-analysis perspective.An understanding perspective is with regard to how well the assets,system functions, and/or security functions with respect to continuoussecurity monitoring are understood. An implementation perspective iswith regard to how well the assets, system functions, and/or securityfunctions with respect to continuous security monitoring areimplemented. An operation perspective is with regard to how well theassets, system functions, and/or security functions with respect tocontinuous security monitoring operate. A self-analysis (orself-evaluation) perspective is with regard to how well the systemself-evaluates the understanding, implementation, and/or operation ofassets, system functions, and/or security functions with respect tocontinuous security monitoring.

The method continues at step 1028 where the analysis system determinesat least one evaluation viewpoint for use in performing the continuoussecurity monitoring evaluation on the system aspect. An evaluationviewpoint is disclosed viewpoint, a discovered viewpoint, or a desiredviewpoint. A disclosed viewpoint is with regard to analyzing the systemaspect based on the disclosed data. A discovered viewpoint is withregard to analyzing the system aspect based on the discovered data. Adesired viewpoint is with regard to analyzing the system aspect based onthe desired data.

The method continues at step 1029 where the analysis system obtainscontinuous security monitoring data regarding the system aspect inaccordance with the at least one evaluation perspective and the at leastone evaluation viewpoint. Continuous security monitoring data is dataobtained that is regarding the system aspect.

The method continues at step 1030 where the analysis system calculates acontinuous security monitoring rating as a measure of system continuoussecurity monitoring maturity for the system aspect based on thecontinuous security monitoring data, the at least one evaluationperspective, the at least one evaluation viewpoint, and at least oneevaluation rating metric. An evaluation rating metric is a processrating metric, a policy rating metric, a procedure rating metric, acertification rating, a documentation rating metric, or an automationrating metric.

FIG. 134 is a logic diagram of another example of an analysis systemdetermining an issue detection rating for a system, or portion thereof.The method begins at step 1040 where the analysis system selects asystem, or portion thereof, to evaluate issue detection processesanalysis the system, or portion thereof. For example, the analysissystem selects one or more system elements, one or more system criteria,and/or one or more system modes for the system, or portion thereof, tobe evaluated. As another example, the analysis system selects one moreevaluation perspective and/or one or more evaluation viewpoints. Theanalysis system may further select one or more sub-sub categories of theissue detection processes analysis sub-category of the issue detection(e.g., detect) category.

As another example of selecting the system or portion thereof, theanalysis system selects the entire system, selects a division of anorganization operating the system, selects a department of a division,selects a group of a department, or selects a sub-group of a group. Asanother example of selecting the system or portion thereof, the analysissystem selects one or more physical assets and/or one or more conceptualassets.

The method continues at step 1041 where the analysis system obtainsissue detection processes analysis information regarding the system, orportion thereof. The issue detection processes analysis informationincludes information representative of an organization's understandingof the system, or portion thereof, with respect to defined issuedetection roles, defined issue detection responsibilities, issuedetection compliance verification, testing of issue detection processes,event detection communication, and continuous improvement of issuedetection processes. In an example, the analysis system obtains theissue detection processes analysis information (e.g., disclosed datafrom the system) by receiving it from a system admin computing entity.In another example, the analysis system obtains the issue detectionprocesses analysis information by gathering it from one or morecomputing entities of the system.

The method continues at step 1042 where the analysis system engages withthe system, or portion thereof, to produce system issue detectionprocesses analysis data (e.g., discovered data) regarding the system, orportion thereof, with respect to the defined issue detection roles, thedefined issue detection responsibilities, the issue detection complianceverification, the testing of issue detection processes, the eventdetection communication, and the continuous improvement of issuedetection processes. Engaging the system, or portion thereof, wasdiscussed with reference to FIG. 121.

The method continues at step 1043 where the analysis system calculatesan issue detection processes analysis rating regarding the issuedetection processes analysis of the system, or portion thereof, based onthe issue detection processes analysis information, the system issuedetection processes analysis data, and issue detection processesanalysis processes, issue detection processes analysis policies, issuedetection processes analysis documentation, and/or issue detectionprocesses analysis automation.

The issue detection processes analysis rating may be indicative of avariety of factors of the system, or portion thereof. For example, theissue detection processes analysis rating indicates how well the issuedetection processes analysis information reflects an understanding ofthe issue detection processes analysis with respect to assets of thesystem, or portion thereof. As another example, the issue detectionprocesses analysis rating indicates how well the issue detectionprocesses analysis information reflects an understanding of the issuedetection processes analysis with respect to system functions of system,or portion thereof.

As another example, the issue detection processes analysis ratingindicates how well the issue detection processes analysis informationreflects an understanding of the issue detection processes analysis withrespect to the security functions of the system, or portion thereof. Asanother example, the issue detection processes analysis rating indicateshow well the issue detection processes analysis information reflectsintended implementation of the issue detection processes analysis withrespect to the assets of the system, or portion thereof. As anotherexample, the issue detection processes analysis rating indicates howwell the issue detection processes analysis information reflectsintended operation of the issue detection processes analysis withrespect to the assets of the system, or portion thereof.

As another example, the issue detection processes analysis ratingindicates how well the issue detection processes analysis informationreflects intended implementation of the issue detection processesanalysis with respect to the system functions of the system, or portionthereof. As another example, the issue detection processes analysisrating indicates how well the issue detection processes analysisinformation reflects intended operation of the issue detection processesanalysis with respect to the system functions of system, or portionthereof.

As another example, the issue detection processes analysis ratingindicates how well the issue detection processes analysis informationreflects intended implementation of the issue detection processesanalysis with respect to the security functions of the system, orportion thereof. As another example, the issue detection processesanalysis rating indicates how well the issue detection processesanalysis information reflects intended operation of the issue detectionprocesses analysis with respect to the security functions of the system,or portion thereof.

The method continues at step 1044 where the analysis system gathersdesired system issue detection processes analysis data from one or moresystem proficiency resources. The method continues at step 1045 wherethe analysis system calculates a second issue detection processesanalysis rating regarding a desired level of issue detection processesanalysis of the system, or portion thereof, based on the issue detectionprocesses analysis information, the system issue detection processesanalysis data, the desired issue detection processes analysis data, andthe issue detection processes analysis processes, the issue detectionprocesses analysis policies, the issue detection processes analysisdocumentation, and/or the issue detection processes analysis automation.The second issue detection processes analysis rating is regarding acomparison of desired data with the disclosed data and/or discovereddata.

FIG. 135 is a logic diagram of an example of an analysis systemdetermining an issue detection processes analysis rating for a system,or portion thereof. The method begins at step 1046 where the analysissystem determines a system aspect (see FIG. 69) of a system for an issuedetection processes analysis evaluation. An issue detection processesanalysis evaluation includes evaluating the system's defined issuedetection roles, defined issue detection responsibilities, issuedetection compliance verification, testing of issue detection processes,event detection communication, and continuous improvement of issuedetection processes.

The method continues at step 1047 where the analysis system determinesat least one evaluation perspective for use in performing the issuedetection processes analysis evaluation on the system aspect. Anevaluation perspective is an understanding perspective, animplementation perspective, an operation perspective, or a self-analysisperspective. An understanding perspective is with regard to how well theassets, system functions, and/or security functions with respect toissue detection processes analysis are understood. An implementationperspective is with regard to how well the assets, system functions,and/or security functions with respect to issue detection processesanalysis are implemented. An operation perspective is with regard to howwell the assets, system functions, and/or security functions withrespect to issue detection processes analysis operate. A self-analysis(or self-evaluation) perspective is with regard to how well the systemself-evaluates the understanding, implementation, and/or operation ofassets, system functions, and/or security functions with respect toissue detection processes analysis.

The method continues at step 1048 where the analysis system determinesat least one evaluation viewpoint for use in performing the issuedetection processes analysis evaluation on the system aspect. Anevaluation viewpoint is disclosed viewpoint, a discovered viewpoint, ora desired viewpoint. A disclosed viewpoint is with regard to analyzingthe system aspect based on the disclosed data. A discovered viewpoint iswith regard to analyzing the system aspect based on the discovered data.A desired viewpoint is with regard to analyzing the system aspect basedon the desired data.

The method continues at step 1049 where the analysis system obtainsissue detection processes analysis data regarding the system aspect inaccordance with the at least one evaluation perspective and the at leastone evaluation viewpoint. Issue detection processes analysis data isdata obtained that is regarding the system aspect.

The method continues at step 1050 where the analysis system calculatesan issue detection processes analysis rating as a measure of systemissue detection processes analysis maturity for the system aspect basedon the issue detection processes analysis data, the at least oneevaluation perspective, the at least one evaluation viewpoint, and atleast one evaluation rating metric. An evaluation rating metric is aprocess rating metric, a policy rating metric, a procedure ratingmetric, a certification rating, a documentation rating metric, or anautomation rating metric.

It is noted that terminologies as may be used herein such as bit stream,stream, signal sequence, etc. (or their equivalents) have been usedinterchangeably to describe digital information whose contentcorresponds to any of a number of desired types (e.g., data, video,speech, text, graphics, audio, etc. any of which may generally bereferred to as ‘data’).

As may be used herein, the terms “substantially” and “approximately”provide an industry-accepted tolerance for its corresponding term and/orrelativity between items. For some industries, an industry-acceptedtolerance is less than one percent and, for other industries, theindustry-accepted tolerance is 10 percent or more. Other examples ofindustry-accepted tolerance range from less than one percent to fiftypercent. Industry-accepted tolerances correspond to, but are not limitedto, component values, integrated circuit process variations, temperaturevariations, rise and fall times, thermal noise, dimensions, signalingerrors, dropped packets, temperatures, pressures, material compositions,and/or performance metrics. Within an industry, tolerance variances ofaccepted tolerances may be more or less than a percentage level (e.g.,dimension tolerance of less than +/−1%). Some relativity between itemsmay range from a difference of less than a percentage level to a fewpercent. Other relativity between items may range from a difference of afew percent to magnitude of differences.

As may also be used herein, the term(s) “configured to”, “operablycoupled to”, “coupled to”, and/or “coupling” includes direct couplingbetween items and/or indirect coupling between items via an interveningitem (e.g., an item includes, but is not limited to, a component, anelement, a circuit, and/or a module) where, for an example of indirectcoupling, the intervening item does not modify the information of asignal but may adjust its current level, voltage level, and/or powerlevel. As may further be used herein, inferred coupling (i.e., where oneelement is coupled to another element by inference) includes direct andindirect coupling between two items in the same manner as “coupled to”.

As may even further be used herein, the term “configured to”, “operableto”, “coupled to”, or “operably coupled to” indicates that an itemincludes one or more of power connections, input(s), output(s), etc., toperform, when activated, one or more its corresponding functions and mayfurther include inferred coupling to one or more other items. As maystill further be used herein, the term “associated with”, includesdirect and/or indirect coupling of separate items and/or one item beingembedded within another item.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may be used herein, one or more claims may include, in a specificform of this generic form, the phrase “at least one of a, b, and c” orof this generic form “at least one of a, b, or c”, with more or lesselements than “a”, “b”, and “c”. In either phrasing, the phrases are tobe interpreted identically. In particular, “at least one of a, b, and c”is equivalent to “at least one of a, b, or c” and shall mean a, b,and/or c. As an example, it means: “a” only, “b” only, “c” only, “a” and“b”, “a” and “c”, “b” and “c”, and/or “a”, “b”, and “c”.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing circuitry”, and/or “processing unit”may be a single processing device or a plurality of processing devices.Such a processing device may be a microprocessor, micro-controller,digital signal processor, microcomputer, central processing unit, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on hard coding of thecircuitry and/or operational instructions. The processing module,module, processing circuit, processing circuitry, and/or processing unitmay be, or further include, memory and/or an integrated memory element,which may be a single memory device, a plurality of memory devices,and/or embedded circuitry of another processing module, module,processing circuit, processing circuitry, and/or processing unit. Such amemory device may be a read-only memory, random access memory, volatilememory, non-volatile memory, static memory, dynamic memory, flashmemory, cache memory, and/or any device that stores digital information.Note that if the processing module, module, processing circuit,processing circuitry, and/or processing unit includes more than oneprocessing device, the processing devices may be centrally located(e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that if the processing module, module, processing circuit,processing circuitry and/or processing unit implements one or more ofits functions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, processing circuitry and/or processing unitexecutes, hard coded and/or operational instructions corresponding to atleast some of the steps and/or functions illustrated in one or more ofthe Figures. Such a memory device or memory element can be included inan article of manufacture.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with one or more other routines. In addition, a flow diagrammay include an “end” and/or “continue” indication. The “end” and/or“continue” indications reflect that the steps presented can end asdescribed and shown or optionally be incorporated in or otherwise usedin conjunction with one or more other routines. In this context, “start”indicates the beginning of the first step presented and may be precededby other activities not specifically shown. Further, the “continue”indication reflects that the steps presented may be performed multipletimes and/or may be succeeded by other activities not specificallyshown. Further, while a flow diagram indicates a particular ordering ofsteps, other orderings are likewise possible provided that theprinciples of causality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

While the transistors in the above described figure(s) is/are shown asfield effect transistors (FETs), as one of ordinary skill in the artwill appreciate, the transistors may be implemented using any type oftransistor structure including, but not limited to, bipolar, metal oxidesemiconductor field effect transistors (MOSFET), N-well transistors,P-well transistors, enhancement mode, depletion mode, and zero voltagethreshold (VT) transistors.

Unless specifically stated to the contra, signals to, from, and/orbetween elements in a figure of any of the figures presented herein maybe analog or digital, continuous time or discrete time, and single-endedor differential. For instance, if a signal path is shown as asingle-ended path, it also represents a differential signal path.Similarly, if a signal path is shown as a differential path, it alsorepresents a single-ended signal path. While one or more particulararchitectures are described herein, other architectures can likewise beimplemented that use one or more data buses not expressly shown, directconnectivity between elements, and/or indirect coupling between otherelements as recognized by one of average skill in the art.

The term “module” is used in the description of one or more of theembodiments. A module implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A module may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a module may containone or more sub-modules, each of which may be one or more modules.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. The memory device may be in a form asolid-state memory, a hard drive memory, cloud memory, thumb drive,server memory, computing device memory, and/or other physical medium forstoring digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A method comprises: determining, by an analysissystem that includes one or more computing entities, a system aspect ofa system for a continuous security monitoring evaluation, wherein thesystem aspect includes at least one system element of the system, atleast one system criteria of the system, and at least one system mode ofthe system; determining, by the analysis system, at least one evaluationperspective for use in performing the continuous security monitoringevaluation on the system aspect; determining, by the analysis system, atleast one evaluation viewpoint for use in performing the continuoussecurity monitoring evaluation on the system aspect; obtaining, by theanalysis system, continuous security monitoring data regarding thesystem aspect in accordance with the at least one evaluation perspectiveand the at least one evaluation viewpoint; and calculating, by theanalysis system, a continuous security monitoring rating as a measure ofcontinuous security monitoring maturity for the system aspect based onthe continuous security monitoring data, the at least one evaluationperspective, the at least one evaluation viewpoint, and at least oneevaluation rating metric.
 2. The method of claim 1 further comprises:receiving, by the analysis system, an input, wherein the inputidentifies at least one of: the system aspect; the at least oneevaluation perspective; and the at least one evaluation viewpoint. 3.The method of claim 1 further comprises: a system element of the atleast one system element includes an enterprise identifier, anorganization identifier, a division identifier, a department identifier,a group identifier, a sub-group identifier, a device identifier, asoftware identifier, or an internet protocol address identifier; asystem criteria of the at least one system criteria being systemguidelines, system requirements, system design, system build, orresulting system; and a system mode of the at least one system modebeing assets, system functions, or security functions.
 4. The method ofclaim 1 further comprises: an evaluation perspective of the at least oneevaluation perspective being an understanding perspective, animplementation perspective, an operation perspective, or a self-analysisperspective.
 5. The method of claim 1 further comprises: an evaluationviewpoint of the at least one evaluation viewpoint being a disclosedviewpoint, a discovered viewpoint, or a desired viewpoint.
 6. The methodof claim 1 further comprises: an evaluation rating metric of the atleast one evaluation rating metric being a process rating metric, apolicy rating metric, a procedure rating metric, a certification rating,a documentation rating metric, or an automation rating metric.
 7. Themethod of claim 1, wherein the obtaining the continuous securitymonitoring data comprises: determining data gathering parametersregarding the system aspect in accordance with the at least oneevaluation perspective, the at least one evaluation viewpoint, and theleast one evaluation rating metric; identifying the at least one systemelement of the system aspect based on the data gathering parameters;obtaining continuous security monitoring information from the at leastone system element in accordance with the data gathering parameters; andrecording the continuous security monitoring information from the atleast one system element to produce the continuous security monitoringdata.
 8. The method of claim 7, wherein the data gathering parameters isregarding at least one of: network monitoring implementation related tothe system aspect; physical environment monitoring implementationrelated to the system aspect; personnel activity monitoringimplementation related to the system aspect; malicious code detectionimplementation related to the system aspect; unauthorized mobile codedetection implementation related to the system aspect; external serviceprovider activity monitoring implementation related to the systemaspect; unauthorized personnel monitoring implementation related to thesystem aspect; unauthorized connection monitoring implementation relatedto the system aspect; unauthorized device monitoring implementationrelated to the system aspect; unauthorized software monitoringimplementation related to the system aspect; and vulnerability scanningimplementation related to the system aspect.
 9. The method of claim 1,wherein the calculating the continuous security monitoring ratingcomprises: selecting and performing at least two of: based on thecontinuous security monitoring data and process analysis parameters,generating a process rating for the system aspect in accordance with theat least one evaluation perspective, the at least one evaluationviewpoint, and the at least one evaluation rating metric; based on thecontinuous security monitoring data and policy analysis parameters,generating a policy rating for the system aspect in accordance with theat least one evaluation perspective, the at least one evaluationviewpoint, and the at least one evaluation rating metric; based on thecontinuous security monitoring data and documentation analysisparameters, generating a documentation rating for the system aspect inaccordance with the at least one evaluation perspective, the at leastone evaluation viewpoint, and the at least one evaluation rating metric;based on the continuous security monitoring data and automation analysisparameters, generating an automation rating for the system aspect inaccordance with the at least one evaluation perspective, the at leastone evaluation viewpoint, and the at least one evaluation rating metric;based on the continuous security monitoring data and procedure analysisparameters, generating a procedure rating for the system aspect inaccordance with the at least one evaluation perspective, the at leastone evaluation viewpoint, and the at least one evaluation rating metric;and based on the continuous security monitoring data and certificationanalysis parameters, generating a certification rating for the systemaspect in accordance with the at least one evaluation perspective, theat least one evaluation viewpoint, and the at least one evaluationrating metric; and generating the continuous security monitoring ratingbased on the selected and performed at least two of: the process rating,the policy rating, the documentation rating, the automation rating, theprocedure rating, and the certification rating.
 10. The method of claim1 further comprises at least one of: determining, by the analysissystem, a system criteria deficiency of the system aspect based on thecontinuous security monitoring rating and the continuous securitymonitoring data; determining, by the analysis system, a system modedeficiency of the system aspect based on the continuous securitymonitoring rating and the continuous security monitoring data;determining, by the analysis system, an evaluation perspectivedeficiency of the system aspect based on the continuous securitymonitoring rating and the continuous security monitoring data; anddetermining, by the analysis system, an evaluation viewpoint deficiencyof the system aspect based on the continuous security monitoring ratingand the continuous security monitoring data.
 11. The method of claim 1further comprises: determining, by the analysis system, a deficiency ofthe system aspect based on the continuous security monitoring rating andthe continuous security monitoring data; determining, by the analysissystem, whether the deficiency is auto-correctable; and when thedeficiency is auto-correctable, auto-correcting, by the analysis system,the deficiency.
 12. A computer readable memory comprises: a first memorysection for storing operational instructions that, when executed by ananalysis system, cause the analysis system to: determine a system aspectof a system for a continuous security monitoring evaluation wherein thesystem aspect includes at least one system element of the system, atleast one system criteria of the system, and at least one system mode ofthe system; determine at least one evaluation perspective for use inperforming the continuous security monitoring evaluation on the systemaspect; determine at least one evaluation viewpoint for use inperforming the continuous security monitoring evaluation on the systemaspect; a second memory section for storing operational instructionsthat, when executed by the analysis system, cause the analysis systemto: obtain continuous security monitoring data regarding the systemaspect in accordance with the at least one evaluation perspective andthe at least one evaluation viewpoint; and a third memory section forstoring operational instructions that, when executed by the analysissystem, cause the analysis system to: calculate a continuous securitymonitoring rating as a measure of continuous security monitoringmaturity for the system aspect based on the continuous securitymonitoring data, the at least one evaluation perspective, the at leastone evaluation viewpoint, and at least one evaluation rating metric. 13.The computer readable memory of claim 12, wherein the first memorysection further stores operational instructions that, when executed bythe analysis system, cause the analysis system to: receive an input,wherein the input identifies at least one of: the system aspect; the atleast one evaluation perspective; and the at least one evaluationviewpoint.
 14. The computer readable memory of claim 12 furthercomprises: a system element of the at least one system element includesan enterprise identifier, an organization identifier, a divisionidentifier, a department identifier, a group identifier, a sub-groupidentifier, a device identifier, a software identifier, or an internetprotocol address identifier; a system criteria of the at least onesystem criteria being system guidelines, system requirements, systemdesign, system build, or resulting system; and a system mode of the atleast one system mode being assets, system functions, or systemsecurity.
 15. The computing readable memory of claim 12 furthercomprises: an evaluation perspective of the at least one evaluationperspective being an understanding perspective, an implementationperspective, an operation perspective, or a self-analysis perspective.16. The computer readable memory of claim 12 further comprises: anevaluation viewpoint of the at least one evaluation viewpoint being adisclosed viewpoint, a discovered viewpoint, or a desired viewpoint. 17.The computer readable memory of claim 12 further comprises: anevaluation rating metric of the at least one evaluation rating metricbeing a process rating metric, a policy rating metric, a procedurerating metric, a certification rating, a documentation rating metric, oran automation rating metric.
 18. The computer readable memory of claim12, wherein the second memory section further stores operationalinstructions that, when executed by the analysis system, cause theanalysis system to obtain the continuous security monitoring data by:determining data gathering parameters regarding the system aspect inaccordance with the at least one evaluation perspective, the at leastone evaluation viewpoint, and the least one evaluation rating metric;identifying system elements of the system aspect based on the datagathering parameters; obtaining continuous security monitoringinformation from the system elements in accordance with the datagathering parameters; and recording the continuous security monitoringinformation from the system elements to produce the continuous securitymonitoring data.
 19. The computer readable memory of claim 18, whereinthe data gathering parameters is regarding at least one of: networkmonitoring implementation related to the system aspect; physicalenvironment monitoring implementation related to the system aspect;personnel activity monitoring implementation related to the systemaspect; malicious code detection implementation related to the systemaspect; unauthorized mobile code detection implementation related to thesystem aspect; external service provider activity monitoringimplementation related to the system aspect; unauthorized personnelmonitoring implementation related to the system aspect; unauthorizedconnection monitoring implementation related to the system aspect;unauthorized device monitoring implementation related to the systemaspect; unauthorized software monitoring implementation related to thesystem aspect; and vulnerability scanning implementation related to thesystem aspect.
 20. The computer readable memory of claim 12, wherein thethird memory section further stores operational instructions that, whenexecuted by the analysis system, cause the analysis system to calculatethe continuous security monitoring rating by: selecting and performingat least two of: based on the continuous security monitoring data andprocess analysis parameters, generating a process rating for the systemaspect in accordance with the at least one evaluation perspective, theat least one evaluation viewpoint, and process as the evaluation ratingmetric; based on the continuous security monitoring data and policyanalysis parameters, generating a policy rating for the system aspect inaccordance with the at least one evaluation perspective, the at leastone evaluation viewpoint, and policy as the evaluation rating metric;based on the continuous security monitoring data and documentationanalysis parameters, generating a documentation rating for the systemaspect in accordance with the at least one evaluation perspective, theat least one evaluation viewpoint, and documentation as the evaluationrating metric; based on the continuous security monitoring data andautomation analysis parameters, generating an automation rating for thesystem aspect in accordance with the at least one evaluationperspective, the at least one evaluation viewpoint, and automation asthe evaluation rating metric; based on the continuous securitymonitoring data and procedure analysis parameters, generating aprocedure rating for the system aspect in accordance with the at leastone evaluation perspective, the at least one evaluation viewpoint, andprocedure as the evaluation rating metric; and based on the continuoussecurity monitoring data and certification analysis parameters,generating a certification rating for the system aspect in accordancewith the at least one evaluation perspective, the at least oneevaluation viewpoint, and certification as the evaluation rating metric;and generating the continuous security monitoring rating based on theselected and performed at least two of the process rating, the policyrating, the documentation rating, the automation rating, the procedurerating, and the certification rating.
 21. The computer readable memoryof claim 12, wherein a fourth memory section stores operationalinstructions that, when executed by the analysis system, cause theanalysis system to perform at least one of: determine a system criteriadeficiency of the system aspect based on the continuous securitymonitoring rating and the continuous security monitoring data; determinea system mode deficiency of the system aspect based on the continuoussecurity monitoring rating and the continuous security monitoring data;determine an evaluation perspective deficiency of the system aspectbased on the continuous security monitoring rating and the continuoussecurity monitoring data; and determine an evaluation viewpointdeficiency of the system aspect based on the continuous securitymonitoring rating and the continuous security monitoring data.
 22. Thecomputer readable memory of claim 12, wherein a fourth memory sectionstores operational instructions that, when executed by the analysissystem, cause the analysis system to: determine a deficiency of thesystem aspect based on the continuous security monitoring rating and thecontinuous security monitoring data; determine whether the deficiency isauto-correctable; and when the deficiency is auto-correctable,auto-correcting the deficiency.